I started using Tailscale, and was amazed at how it was able to make connections directly to my devices on different networks, even behind Carrier Grade NAT on mobile data. (You can read more about Tailscale here)
The NAT traversal tech Tailscale uses is pretty nifty, and I was interested in using their techniques to solve more problems.
For example, file sharing! File sharing is such a hard problem, especially with the size of the photos and videos we take. While messaging apps like Signal do allow sharing of such files, there are always size constraints on them, and for photos and videos, there is also compression to contend with. Compressed photos do not look nearly as good for archiving purposes, and just plain viewing!
Services like Google Drive are good, but not as private. You reveal to Google not only the file(s) you want to share (unless you encrypt them before uploading), but also have to consent to automatic scanning of pictures for illegal materials. You could encrypt the files and then upload, but that’s not nearly as seamless as using a messaging app to send compressed versions. So, what do we do?
It’s kind of fair: we’re consuming a third-party’s network bandwidth and disk space to do our work. There are bound to be some compromises. But, on the other hand, we have some really impressive network speeds right at home, and often times the only real time it’s strained is video (or game) streaming. How’s about we change that and use just our resources to transfer our files?
Note: Tailscale also has a file transfer tool built into it’s clients. However, what’s the fun in using that? Also, for casual/impromptu sharing of files, this isn’t exactly ideal since you’d need to invite the other party’s devices on your network as well.
This is an excellent article from Tailscale’s own blog where they explain the various tricks they use (and don’t use) which allows us to make direct connections.
The most basic idea is this:
Firewalls between you and your destination typically only trust connections from you. You have to initiate connections in order for the firewall to go, “Ah, since you want to go to this destination, I’ll trust it and allow packets from that destination to you from now on”.
However, if both source and destination are behind NAT, that means:
They don’t know what they actually look like on the public Internet. They only know their local network IP address and port, which is next to useless for connections over the Internet
They don’t know what the other party looks like on the public Internet, so there’s no way (for now) that they can even begin to communicate
The other party likely doesn’t have an open port for them to connect to. The port, for them, will only open when the other party knows their external IP and port.
So how do we solve this?
Contact a STUN server, which is a glorified way of saying “Use the same socket you want to use for communicating with the other party to communicate first with a server on the Internet which tells you what your public IP and port is”
This can be solved in a myriad of ways, from basically typing the IP and port into the server as well (for testing purposes, or if you just want 1:1 communication), to having a “control plane”, as Tailscale calls it, which is just a small server that passes such small messages between machines. While this is centralized, it doesn’t carry much info and is pretty cheap compared to the cost of communicating entirely through a relay (especially if you’re running a file sharing service).
This is where we exploit a loophole.
Note from before, the firewalls trust incoming connections from other parties only if they have seen you contact them first. So, why not just “contact” them? Sure, the message probably won’t go through to the other party (what with their firewall blocking you), but it only has to reach your firewall for it to go, “OK, so we’re trusting this outgoing IP. I’ll allow incoming connections from it to you.” At the same time, have the other party do this for you, and after a few short moments, both parties (behind NAT) can now communicate directly with each other!
NAT traversal typically involves using UDP sockets to negotiate a connection. This is because TCP itself adds a layer of complexity to an already complicated procedure, and certain capabilities are not possible without kernel modifications.
So, we’ll need to write our own mini-TCP replacement on top of UDP in order to reliably pass not just a file, but related metadata, like the size of the file as a way to confirm that a user really does want to download a file,or (in the future) a public key that serves both as a unique identification for a user and a way to secure the connection so no one knows what file you’re sending.
This is also the perfect project for me to help learn Rust, as it is a low-level, high-performance application that should also be safe as possible to use due to its sensitive nature.
While I had thought of working on the NAT traversal and file sharing aspects of the project separately, I eventually decided to start working on file sharing first and then build the protocol (including NAT traversal) around it. I did it this was as I needed some time to get used to Rust and how to get around the borrow checker.
At the end of this exercise I just wanted a proof of concept: something that would actually do the least complicated version of what I want to achieve and set the groundwork for solving a more complicated problem.
Building a basic file transfer protocol was a bit harder than I had thought it would be. Right now, my unnamed file transfer protocol is just a straight rip off of TFTP: send the file in appropriate packets, wait for an ACK and re transmit if you never get the ACK. There are some differences, such as adding some checking for authorization (which will help out when encryption support is added), sending file size in order to truly confirm the user wants the file, and a slightly more complicated data transfer scheme which goes like this:
Client sends a request to get data from offset X
Server responds by getting data from offset X to X + DATA_SIZE or till the end of the file, whichever is smaller.
Client sends ACK for the same, increments X by however much data it had received, and requests the next chunk of the file if necessary.
This seems slow for a number of reasons:
It is still sequential even if we do have some concurrent requests going on
Too many round trips: just get rid of the ACK for the packet we receive and directly ask for the next chunk or send a resend request
I’ll need to work on this in order to get better throughput for the program.
After I had gotten some level of file transfer capability integrated, I just added a step before the file transfer would be initiated to just blindly have both parties send packets to each other and hope that they had also gotten the message. There is no reliability check here other than just relying on dumb luck that out of maybe 5 packets, at least one could make it to the other party. A time delay of 5 seconds is added both before and after this little procedure in order to sync the entire thing together.
At first, my file transfer protocol seemed to work over the Internet almost perfectly for small files. It was upon sending bigger files (even a couple of megabytes would do the trick) where I would see the protocol never recovering from lost packets. In fact the rudimentary protocol I had arrived at in the previous section was the result of a whole bunch of experimentation and research after integrating NAT traversal, as there really is no better way to test reliability than routing all my requests through a massive, busy network full of a huge volume of packets being routed to wherever they need to get to.
This proved to be the correct strategy, and would prove to be the final test I would put every iteration of my protocol through.
There are several limitations with the current code:
Only one file can be transmitted at a time.
No real concept of directories exists either.
The file to be transmitted is loaded completely into RAM before being transmitted. Now, for images and some videos that is tolerable, but what about very big files, like ZIP archives of photo albums from many years? We can’t just expect to load all that into RAM, and it is also not great for performance on low-end machines and mobile devices. Something needs to be changed to avoid such a costly maneuver to be required, such as utilizing mmap() to address content on disk as if it were just in RAM, though this may come with performance penalties. This can also be done on client side to write the file to disk as well, as right now the file is assembled in RAM and then written to disk after being received in its entirety.
Only one client is really supported by the server since there exists no way to initiate a connection to another client by server right now. This is planned to be fixed by building a small control plane which will just use TCP to communicate with both clients and servers and help them connect to each other. This is the only machine which will need to be on the public Internet, and so will cost extra resources to run. So, for now, this is on a backburner in order to get the rest of the problems solved.
No encryption. In this day and age this makes it a deal-breaker for actual production usage, and must be rectified in order for other people to actually derive value from the project. Also, no authorization control really exists; I have just added a dummy value for the AUTH field for testing purposes.
Slow transfer speed, even with very little lost packets. The throughput is just very low, and the program still feels like it is making requests sequentially. I will need to investigate further and eliminate more bottlenecks to really saturate the network link.
At this moment, I have a CLI program that asks for the client/server IP:port upon launch, attempts to trick the stateful firewalls to allow a direct connection, and then initiates a file transfer. Truly the most minimal version of my problem is solved, and it lays the groundwork for the journey ahead.
Still a long ways to go to solve this problem, but I do have a couple ideas as to how I can do it.
You can check out the code here. For future allied projects under this common goal, check out this link
That’s all for today! Bye now!
This website was made using Markdown, Pandoc, and a custom program to automatically add headers and footers (including this one) to any document that’s published here.
Copyright © 2023 Saksham Mittal. All rights reserved. Unless otherwise stated, all content on this website is licensed under the CC BY-SA 4.0 International License