The question that popped to my head was why FreeBSD? I stopped using it for anything back in 2006. Since much of this tuning/debugging/analysis relates to the OS/kernel shouldnt the article start there?
That decision was made long before I joined Netflix. My understanding is that the initial decision was made due to the license. After that, our team attracted a large number of FreeBSD committers, so it just kept its momentum.
I've done kernel work in both OSes, and I'm pretty much agnostic myself. I find the biggest drawback to FreeBSD is hardware support. Netflix has actually driven a fair amount of vendors to provide FreeBSD support due to our using FreeBSD.
I will say that FreeBSD has a few things that Linux doesn't, specifically async sendfile. This ends up being far more efficient than either a thread pool, or aio daemons for handling IO completions. Async sendfile is also one of the things that drove us to implement TLS encryption in the kernel, and is the foundation that pretty much our entire stack rests upon.
> Netflix has actually driven a fair amount of vendors to provide FreeBSD support due to our using FreeBSD.
Yes, and the rest of us FreeBSD users greatly appreciate you for that! Please keep using FreeBSD and pushing these vendors and this household will keep throwing 2 x $10/month at Netflix. :-)
I have heard it described like this. Suppose you want to hire some real kernel talent.
So you go to a Linux conference looking for people. When you get there the conference is crowded with idiots, advertisers, and companies. No way to find the smart people.
Now you got to a FreeBSD conference. It is almost empty but the majority of the people know what they are doing. You only need to be interesting, have a clue, and buy a beer to get all the attention you want. ;-)
From an objective standpoint, the Linux stack is actually better in a lot of ways. They are generally much, much, much tighter in terms of cache misses and per-packet costs. They have a lot of features that FreeBSD doesn't, and a lot of innovation happens there (BBR).
And they have a huge number of very smart people working on it. I've had the privilege of working with some of them at Google, and have nothing but respect for them.
From a business perspective, I think it's easier to pick up *BSD talent. Although the pool is smaller, the pool is more talent dense. If you can afford to not care about the kernel (serverless etc) Linux is fine, if you have to care about the kernel and hardware I'd pick FreeBSD 100% of the time. Also much easier to influence overall project direction and get patches integrated.
> The networking stack has also handily beat Linux for as long as I can remember.
None of the TOP500 use FreeBSD [1]. I would have thought this would be a primary consideration for a supercomputer. Most of them are using Linux (99.6%!).
For internal purposes like these, they're pretty much indistinguishable. Remember that the GPL only compels you to distribute the source to whoever you gave the binaries, not to everyone else. And distribution inside your own organization doesn't count.