It sounds very promising. In-line TLS is the holy grail for us, since it would cut our memory bandwidth usage by over 40%, and memory bandwidth is our biggest bottleneck.
However, I'm not very bullish on just plain crypto accelerator mode. We've tried other vendor's accelerators, and while they save some CPU, memory bandwidth is the real issue for us. The data still needs to be encrypted, so it still needs to be read from memory by DMA rather than a CPU read, and written via a DMA write rather than a CPU write.
(BTW: Huge kudos to Netflix for driving vendors to support
FreeBSD)
Unfortunately exposing my nativity on both NICs and TLS, I've
never understood why NICs couldn't support the crypto in a
way that it can just send the packet straight to the wire
after encryption. Is there some deep (unfixable) reason in
the TLS/TCP protocol or just a lack of foresight in the NIC
design?
What makes in-line TLS difficult is that the NIC has to understand the TLS framing and the TCP segmentation for TSO. On top of that, what do you do if you have a re-transmission. The NIC has to re-download the entire 16KB TLS frame to re-send just a 1448b portion of it. So there are some challenges in terms of being able to find the entire TLS record, making sure to reference the entire thing so that it is all still there, etc. The "1 mbuf per TLS record" work I've done will make that a bit simpler, I hope.
>download the entire 16KB TLS frame to re-send just a 1448b portion of it.
The bigger thing, is that TLS is far from ideal for video content, and TCP transmissions in general.
It is much better to incorporate a degree of redundant coding, and tolerance to lost packets than rely on re-transmission.
As you deal with a real-time content, there is a no real need for flow control. You either get packets in time, or your video feed cuts of.
Another moment, for any genuinely live transmission, multicast beats down everything for efficiency, but why it is so hard to run multicast over open Internet is a discussion on its own.
Given that packet loss on the Internet tends to be very low, like 0.1%, retransmission is a lot more efficient than coding. Being able to build up your buffer faster than real time also helps with burst losses.
why NICs couldn't support the crypto in a way that it can just send the packet straight to the wire after encryption.
That's how the Chelsio T6 works. On the transmit path the NIC needs to perform encryption, TLS framing, and TCP/IP/Ethernet segmentation in that order; on receive it needs to perform TCP reordering and such before decrypting. Most NICs just don't want to add that much functionality.
Thanks, that's looks pretty good. I wonder if it will work with TLS 1.3 as well (from my limited understanding it looks like it might).
Here's dreaming of a future where ED25519/Chacha20/Poly1305 is supported in hardware (it's much cheaper for clients that doesn't have AES so I want servers to use it).
However, I'm not very bullish on just plain crypto accelerator mode. We've tried other vendor's accelerators, and while they save some CPU, memory bandwidth is the real issue for us. The data still needs to be encrypted, so it still needs to be read from memory by DMA rather than a CPU read, and written via a DMA write rather than a CPU write.