Serial port over internet, peer-to-peer and through both firewalls

[voltsandjolts]

I would like to connect to a RS232 serial device on the other side of the world.
I would also like to do this through both corporate firewalls without changes to either of them.

Ideally, I would like to send a client an 'intelligent adapter' which they plugin to our product serial port and to their network (lets assume internet access allowed without logon/authentication, DHCP sets up gateway address etc.). A similar adapter or virtual comm port is used at the other end. No firewall changes (opening ports) at either end.

So, I am thinking, "UDP hole punching with rendezvous server" kind of setup. Similar to, say, VOIP or Dropbox but with serial ports at the end.

A pure software solution might be http://www.flexihub.com/

Are there any other options out there?

[StuUK]

http://www.taltech.com/tcpcom


http://www.eltima.com/rs232-over-ethernet/

maybe?

[voltsandjolts]

http://www.taltech.com/tcpcom
http://www.eltima.com/rs232-over-ethernet/
maybe?

Maybe, but I doubt it. They don't give much technical explanation nor do they mention firewalls at all.
They do talk about connection over internet but that is probably achieved by opening ports.
I'll download trial versions if nothing else turns up.

[StuUK]

The assumption would be that firewalls are configured appropriately for the ports/addresses that are trying to communicate which (if over the internet) would be down to the local and remote end firewall configurations.

[voltsandjolts]

The assumption would be that firewalls are configured appropriately for the ports/addresses that are trying to communicate which (if over the internet) would be down to the local and remote end firewall configurations.

Thanks for your input but you didn't read as far as the second line in my original post:

"I would also like to do this through both corporate firewalls without changes to either of them."

[StuUK]

I did which is why I mentioned the assumption that the firewalls are configured appropriately, i.e. fairly relaxed on source/destination addresses and ports.

Any solution you look at will inevitably require the local and remote ends to be configured with a port to use as a minimum. If you can utilise a port already open and the source/destination addresses are ok then you're good to go.

If the corporate firewalls are pretty locked down then it'll be an interesting ride....

[voltsandjolts]

Peer-to-peer communications do not necessarily require open IP ports (as many popular applications demonstrate).
I am specifically looking for a solution which does not require open ports.
Thanks.

[StuUK]

Popular 'Peer to Peer' applications generally use a signalling server to negotiate connection details. I'm assuming that you require something that you 'plug' in both ends and communicate directly, presumably over the internet, and don't involve a third party service/server but I can see I'm not helping so i'll duck out.

[Emo]

Hi,

I think the virtual solution from eterlogic might be a solution
http://eterlogic.com/Products.VSPE.html
I use it for several years now on linux and windows.

regards,

Eric

[mariush]

If the transfer rates are relatively small you could make your product completely passive as in don't allow anything to connect to device, let the device send data to a web server and request incoming data from same web server periodically.
Give each adapter a unique Id ( which can me stored in the microcontroller's eeprom memory or something like that) and the micro connects to the website and does a request that looks something like this GET /receive?uid=12345&lastpacket=24354  wherethe uid is the unique id of the device and lastpacket is the unique id or time stamp of the last packet of data the server sent to you, so that in case connection is suddenly broken , the adapter can reconnect and resume transfer of data without losing anything.
in theory the connection can be kept open up to minutes or hours, because that's how online radios typically work these days, so ISPs shouldn't interrupt you. But in the worst case that you have a bad router or switch that kills connections or they freeze after some time, the micro can close connection and create another request to the remote server using the last packet id as resume point.

As for sending data to user, this could also be done using a simple POST http request to the web server with data encoded in a variable or optionally declaring the data as a file upload of unknown/very long size and just killing the connection when you feel like stopping transferring messages. you can use even a GET request encoding the data to send in a parameter in the URL, but you'd have to be careful about length of URLs (less than about 4K would probably be safe)

On the server side, the incoming and outgoing data could be kept in RAM using a memory / disk backed database (think memcached but with no loss guarantee, keep data in ram and delete packets as soon as they're transferred and no longer needed to be stored)
The problem with this system is the added latency between device and user - you get the latency between device and server and then from server to user and matching but otherwise with html 5 and javascript these days you can make some kid of terminal fairly easily.  Also, if the user turns out to have a static IP or that your application can break through his firewall and your server can connect from outside to the user's computer inside a network, it could optionally instruct your adapter to talk directly to that user's ip:port pair until disconnected, and then fall back to default device-server-user behavior.
Think o all "shoutboxes" that are on various forums which pretty much work on the same concepts.
ps and html 5 also has web sockets which makes chat back and forth very easy. 

ps. my keyboard died, using a temporary shitty one with very hard to press buttons so in my posts i may miss some letters, sorry if they look bad or are hard to read

[voltsandjolts]

I was looking for an off-the-shelf solution but maybe it could be a back-burner project.

I was hoping for peer to peer (using initial 3rd party/rendezvous server) just to keep latency down but it seems P2P is not always possible behind firewalls - depends on the setup:
http://stackoverflow.com/questions/23176800/whats-so-hard-about-p2p-hole-punching

Passing all data through a cloud webserver might be the more reliable way to go, albeit a little slower. I guess there are many ways to tackle that but websockets would seem the modern way, perhaps a raspberry pi at each end using python pyserial/websocket to cloud server which would connect the two sockets together.

Latency on this websocket server seems OK:
https://www.websocket.org/echo.html

[voltsandjolts]

I'll give www.flexihub.com a try.

Note to future self:
I think a DIY solution could made out of python + http://autobahn.ws/python/ + http://crossbar.io/
Example serial port over websockets here:
http://tavendo.com/blog/post/arduino-yun-with-autobahn/

[IanJ]

Hi,

Been many years, but I am sure I used Lantronix serial servers to do this, and configured them to use port 80. Tested city to city and through Cisco routers/firewalls.
Devices were Lantronix MSS1 & MSS4 but I am sure there will be new models now!

Ian.

[voltsandjolts]

Thanks Ian but it looks like all their current serial servers require ports to be opened for internet comms.

[Jeroen3]

You can try an eWon. However, it looks like you need two m2m and not h2m.
Theoretically it should be possible to connect two eWon sites together. But you'll have to ask them.

You get an OpenVPN network in their cloud to which you can connect with your laptop. The uplink device in the machine also connects to this cloud. Not firewall changes required, except vpn pass-trough I think.

So, I am thinking, "UDP hole punching with rendezvous server" kind of setup. Similar to, say, VOIP or Dropbox but with serial ports at the end.

You have two problems with that:
1. How do the devices know each others WAN IP.
2. A firewall surely does not accept random UDP frames except from the destination of the outgoing UDP frame. Or does it?
It's what UPnP was invented for, right?

[Aeternam]

Thanks Ian but it looks like all their current serial servers require ports to be opened for internet comms.

Ports are an integral part of the communication infrastructure of a computer network, there's no way around them. https://en.wikipedia.org/wiki/Port_%28computer_networking%29

There is no (easy and/or legal) way you will be able to communicate with a device or service behind a firewall if the firewall is configured to block the port through which you want to address your device.  HTTP (the protocol used for web browsing) for instance uses port 80. Now usually port 80 is open on the corporate firewall for people to be able to check their webmail or shop online and you could use it to communicate with your device. But if the firewall blocks that port, you can't surf the web and you won't be able to reach your gear.

In your opening post you make the assumption of an "open internet access and default gateway config." If this is the case (on both ends) you will be able to reach both of your devices through port 80. You would still need to know their public IP address in order to connect to them.

Be advised also that network admins (especially the ones in charge of large corporate networks) do not like traffic on their network that they don't know about. You might want to talk to them before hacking away 

[Jeroen3]

There is a fundamental difference between source port 80 and destination port 80.
To view a website, you use destination port 80 on [url]. To do that you use source port anywhere from 49152–65535.
Blocking port 80 will not prevent any user from accesing webmail or webpages over http.

Firewalls usually do not care about destination ports. Unless they are specific to some protocol. Such as torrents.

[nuno]

You'll need a server between both and outside both local networks, where all data has to go through (not only for initial connection), I see no way around it. Maybe a PHP/.NET/servlet/whatever web page could be used thus avoiding the more onerous way of having a full server under control. You might get away with some free IM way, not sure what's available today.
It would be a lot easier if your system could accept a connection on one side (such as on your company's side).

[voltsandjolts]

@Aeternam
As Jeroen3 says, you are misunderstanding how ports work.

You can try an eWon. However, it looks like you need two m2m and not h2m.
Theoretically it should be possible to connect two eWon sites together. But you'll have to ask them.

Thanks, I'll have a look at that.

You have two problems with that:
1. How do the devices know each others WAN IP.

That's the job of the rendezvous server, to initiate the connection, tell each end the IP address of the other.

2. A firewall surely does not accept random UDP frames except from the destination of the outgoing UDP frame. Or does it?
It's what UPnP was invented for, right?

That's what UDP hole punching is for but, unfortunately, some networks may disallow it:
http://stackoverflow.com/questions/23176800/whats-so-hard-about-p2p-hole-punching

UPnP won't work in corporate network.

[voltsandjolts]

@Jeroen3
Ooooo, this might be the jackpot. It even says Firewall Friendly! Now to test if I can get OpenVPN outbound from here....

[CaptCrash]

There is a fundamental difference between source port 80 and destination port 80.
To view a website, you use destination port 80 on . To do that you use source port anywhere from 49152–65535.

The source port range depends on the client software, recent versions of windows use the ranges you have listed, older versions use 1025-5000, other operating systems/IP implementations will use other ranges of source ports.
https://support.microsoft.com/en-us/kb/929851][/b]. To do that you use source port anywhere from 49152–65535.
[/quote]
The source port range depends on the client software, recent versions of windows use the ranges you have listed, older versions use 1025-5000, other operating systems/IP implementations will use other ranges of source ports.
[url]https://support.microsoft.com/en-us/kb/929851

Blocking port 80 will not prevent any user from accesing webmail or webpages over http.

If you mean, blocking a source port of TCP 80, then yes you are correct.
If you mean, blocking a destination port of TCP 80, then no you are incorrect.

e.g. If you add a rule that blocks or drops all traffic with a destination of TCP port 80 and place this rule at the top of the rules list (giving it the highest priority), then you will not be able to browse to any external HTTP (TCP port 80) traffic via this firewall.

Firewalls usually do not care about destination ports.

For outbound traffic, Firewalls only care about destination ports (you can also limit source ports but its not typical).  What you have written is entirely backwards.

Unless they are specific to some protocol. Such as torrents.

There is nothing different about one protocol (port) from another from a firewalls perspective, connections are allowed or not (there are some special cases, like established sessions, responses from other ports like with FTP, but generally its fairly straight forward).
Torrents are in no way a special case, they have the exact same limitations/capabilities as any other protocol in that they need to be allowed through the firewall.
This requires either explicit rules on the firewall (or very broad rules that just allow everything out) or via some automatic process like UPnP).

[CaptCrash]

@Aeternam
As Jeroen3 says, you are misunderstanding how ports work.

Sorry but Jeroem3 has it backwards.
@Aeternam got it correct in terms of how ports work and how any reasonable network admin would approach configuring a firewall.

[voltsandjolts]

Aeternam is misunderstanding the term "open port", or at least what he has written suggests that.

[David Hess]

Assuming that the firewalls allow outgoing UDP, I would look into using preconfigured OpenVPN network appliances on both ends which automatically connect to a central OpenVPN server so the network endpoints then see each other as local.  TCP will will work for this as well although probably not quite as well and if desperate and port 80 is not through a proxy, outgoing port 80 could also work.

It has been a while since I have used OpenVPN but I think this is possible to do this without knowing the endpoint IPs.  The endpoints just need to know the central server's IP.

[wine+dine]

There are many possible ways in which firewalls are configured.  If you have to work with a corporate firewall on either side that you don't know about in detail and can't have modified, this can be a hard problem.

A simple firewall in the past may have just filtered connections based on addresses and ports, e.g. "from internal to anywhere with destination port 80 except those on the blacklist".  In such a case, a reflector server can listen on port 80, accept connections from both endpoints, identify them and relay data from one to another.

"Next-generation" firewalls such as the one I manage for my employer perform deep inspection of the traffic and know about the lowest level details of application-layer protocols, not only network-layer.  What that means is that the port-80 reflector traffic would be blocked even if connections to web servers on port 80 are allowed, because your data would not look like legitimate HTTP. 
Vendors such as Palo Alto have turned this sort of inspection into an art form, with decoder logic updating every few days to match developments, and rule granularity available down to allowing specific functions on a service but not others, e.g. Gmail yes but attachments no. (The decoding is done at gigabit+ speed on FPGA-based hardware, BTW.)

The web sockets idea is a good one but keep in mind that the above means there is no absolute guarantee it will work even if both sides can connect to your server.