Why not run a regular FTP server and have your application periodically look for new files to process? For horizontal scaling, you just take a distributed lock on the file name.
That honestly sounds more complicated. FTP isn't that difficult of a protocol, especially if you only need to support one known client, you can take all sort of shortcuts.
If you deploy an existing FTP server, and _then_ integrate with it at the filesystem level you now have two components, and your sysadmin requirements grow. Now you gotta administrate an FTP server that's probably written for classic UNIX single server usage, gotta handle filesystem permissions, gotta somehow hook up your distributed locks to the filesystem, sanitize filenames for your chosen filesystem.
Honestly filesystems suck, there's so many gotchas from a security perspective, when all you really want is to pipe binary data in this side, and out the other side.
I implemented an IRC bot in a few hours in javascript one day. Those classic IETF text based protocols are actually really fun and easy to implement, especially in a language that makes strings safe and easy (i.e. not C).
I could easily see figuring out all the deployment concerns around integrating with an existing FTP server end up taking way longer than just integrating the subset needed for this use case.
Strings in c aren't that bad with stuff like bstring or glib. Just gotta be careful with deallocations so you don't get leaks. Much better than all the security issues with std strings. I mean not anywhere close to python/ruby/javascript easy but it's not as bad as HN likes to declare it.
Active mode is pretty weird. Coordinating a single client across two ports sounds difficult to me, but I've never implemented it. If that's not a difficult protocol, then what is?
Cordinating a client across two ports sounds trivial compared to for example properly implementing client and server versions of IMAP search commands when no client or server follows the specification.
You will have similar issues with other protocols as well. I have seen physical bandwidth limiters, windows drivers, corrupt winsock LSPs etc mess with IMAP traffic. I have seen middleware replacing SMTP commands sent from a client to server for no good reason, such as replacing "EHLO hostname" with "EHLO **".
The lock would be handled by something like Redis or a DB.
But yeah, if you only need to support one client, I can see the reasoning. It would never have flied at any of the places I've worked, though, having to support tons of clients.
I had considered this, but decided against this for a couple of reasons:
- Scaling requirements are relatively low. Even though we're dealing with 10 thousands of devices, the amount of firware updates at a given time to those devices is minimal. Our main scaling challenges are around OCPP over websockets. Story for another day.
- I have bad memories of ProFTPd etc buffer overflow exploits.
- I wanted something simple that could bridge between FTP and our cloud persistence (MongoDB and Cloud Storage).
- I found this Node.js library that I since then forked: https://github.com/autovance/ftp-srv - The great thing about this library is that it allows a quick implementation of a custom filesystem.
- For Kubernetes pods the file system should really be treated as a /tmp - which we are doing.
- When a charge station connects, the FTP username/password is a temporary generated set of tokens that is checked against our MongoDB.
Essentially, I'm using FTP as a throwaway here.
If you think through this you can imagine it would be quite a lift to accomplish this with an existing FTP server.
They describe why, tons of edge cases from poorly implemented ftp clients, they needed full control over the protocol to handle weird edge cases because of multiple embedded clients.
Not necessarily so. The history of FTP servers is ridden by bugs with practically no exceptions. At some point some folks decided they finally implement a bug-free implementation and even dared to call it "Very Secure FTPd." Needless to say, it turned out it has bugs, too.
As most of these bugs were related to buffer overflows and similar issues, implementing a new FTP server in a safer language is not such a bad idea, and today's JavaScript is efficient enough to make it a reasonably well-working implementation. I pity the author though for the bugs they encounter and workarounds that will need to be implemented.
Exactly my reasoning. See my comment above for more info.
I'm making the maintenance of this less painful by doing a hacking/debugging session with manufacturers once a month where we hook up many devices and fix issues. After addressing most edge cases fewer are coming up now (despite a relentless stream of new cheaply manufactured devices)
Why not run a regular FTP server and have your application periodically look for new files to process? For horizontal scaling, you just take a distributed lock on the file name.