If you want more technical details, we've put up a webpage back then, and published a background paper on those vulns:
https://nostarttls.secvuln.info/
You explicitly call out imap and other email protocols. Can this be applied to LDAP?
LDAP (think Active Directory if you are short of imagination, and/or experience).
A lot of connections use STARTTLS on port 389 instead of full on explicit TLS on port 636. Then there are the other two ports for the "global catalogue" which I think is basically a Win NT style domain flat lists for users and groups on 3268/tcp and 3269/tcp.
I've always had my suspicions about STARTTLS but it looked quite convincing to a sysadmin and was always encouraged by the sort of people who use terms like: "best practice". I'll start dumping it from now on. This will take a while.
I’m not the writer of the paper, but I do remember STARTTLS being called out as an issue when I was previously working on setting up an ldap directory.
NBD[1] is another protocol that uses STARTTLS, although as a protocol it's not commonly exposed on the internet (and in fact encryption is not commonly used either since most people are interested in pure performance). We've tried to mitigate the worst effects by reducing the types of message that are permitted before you upgrade to TLS to the bare minimum.
STARTTLS command buffer smuggling was already a known issue long before 2014. In particular, the deluge of CVEs for MTAs started in early 2011, and it was obvious from the nature of the issue that it wasn't limited to SMTP, but rather pertained to lower level I/O interfaces common to network facing servers generally.
I worked on a project that filed bug reports about this 19 years ago. We blocked the clear text port, full stop. Can't count on users to config things correctly, even programmers.
I don't mean to be difficult, but in the spirit of being forthcoming:
"Called it!" posts really turn me off. You were right before, that's cool. What else can you add to a potentially interesting conversation? Bring your audience along for the ride..
I'm much more interested in any new insights you have on the matter!
Best wishes and take care <3, and again, I'm sorry, I wish I had the perfect words to connect with you.
Their blog post from 2014 does seem to "call it" quite succinctly, with a rather beige background.
This is a called it post with evidence. Why not read it! The current post has only just been ... posted. I'm sure Andrew will have an angle (if he can be arsed) but he does get to call it: "Called it".
Words are never perfect unless they are conjugating to a particular tense 8)
To this day plenty of documentation refers to STARTTLS as being the preferred way of configuring your email client. This is _clearly_ insane in the face of implicit TLS existing. I've never understood why STARTTLS is recommended over implicit TLS.
I think it was because of corporate firewalls. Unencrypted ports for POP3, IMAP, and SMTP were widely known and allowed where necessary, but their implicit-TLS counterparts were often blocked and IT departments were slow to adapt. STARTTLS helps you smuggle an encrypted tunnel over a well-known port.
Nowadays, it's often the reverse. Many clouds block outgoing port 25 by default, and sometimes even 587. But almost nobody blocks port 465 -- the assumption is that if you're using 465, you must be authorized to use that server -- so that's the port you must use if you want to send emails by SMTP from a typical cloud server.
There was a sense of "wasting a port". A modern Linux /etc/services has only 200 or so reserved TCP ports (out of a possible ~50k) so that fear might have been overblown.
I suspect the bureaucratic overhead of needing to go to IANA to reserve a new port might have had a chilling effect. See:
This makes sense, because perhaps the world's most important email client that still costs money to buy (Outlook) used the terms SSL and TLS like that for about a decade. These days the toggle says "SSL/TLS" versus "STARTTLS", I believe, or they may have even dropped the toggle completely to just auto-detect the protocol based on the port number.
Using correct terminology would've led to tons of unnecessary helpdesk calls because Microsoft messed up its naming scheme.
Many cloudy environments end up economizing on IP addresses. That means that there will be loads of neighbours, not all of them will be your mates. Once you have found a way in via another method, you now have this weapon to deploy. STARTTLS is massively used to secure internal comms as a (cough) "best practice". The article notes email related protocols but AD via LDAP is often accessed via port 389 with STARTTLS.
You may not be familiar with Hanno - he's been around for a while and knows what he is on about.
I'm not sure which real world you inhabit. This is quite horrendous.
(I'm not sure why "implicit" is used instead of "explicit" for full on TLS over STARTTLS. I suspect English as a foreign language).
"Explicit" is to be understood as asking for STARTTLS on OSI Level 7 instead of Level 4. If the client cannot establish an encrypted connection, the attempt is aborted in the first place. Of course, there is the hazard as in HTTPS that an MITM attacker might somehow cause a downgrade to the unencrypted protocol variant.
as a civilian, if cloud compute server conditions internally result in exposing some mid-transaction packet insertion problem as a real thing, AND that in turn justifies some massive re-deployment of new network drivers stacks to the detriment of just about everybody's gray hairs and finite hours in the day, then actually screw that, with emphasis.
Your keyboard should have at least two shift keys. We are generally all civilians (both of my parents were soldiers - I'm not but I did love growing up in West Germany, funny old world).
This is yet another CVS and as I happen to be the MD of an IT company, I take it seriously. It is a bit of a pain having to patch a lot of systems every month and fiddle with firewall rules in response to notes and more besides.
I don't intend to feature in an el Reg or HN article. I've been lucky so far.
> STARTTLS is massively used to secure internal comms as a (cough) "best practice"
Well, I've seen it be a "best practice", because of some legacy component that only supported ldap with starttls or plaintext. It wasn't so much "starttls is great" as "starttls is the best we can do right now".
With the amount of air conditioners and coffee makers (et al.) connected to WiFi I think lapses in correct separation is more frequent than anyone would be willing to admit. Maybe not too practical to hijack a TCP stream, but still worth to remember if anyone claims to be protected by NAT.
If you can redirect the victim to a proxy via DNS poisoning you don’t need this attack you can simply pretend to be the mail server; not only that but it doesn’t actually allows you to execute this specific attack.
How this works is that you inject the commands into the plain text of the victim which requires packet racing or some other local network attack that allows you to take over the TCP stream.
How this works is that the victim sends from:, creds, to:, email content but you somehow inject starttls, from:, creds:, to: a head of it which forces the entire message from the victim to be treated as data fully so the SMTP commands are disregarded which means you will be receiving an email with their credentials.
I highly suspect that SMTP without starttls will work in the same manner, but perhaps the opportunistic encryption gives more time for this attack to be more effective.
Every other scenario requires you to be able to MITM the traffic which is pretty much already gameover.
I honestly can’t really think of a scenario where you could launch a network attack that would allow you to inject yourself in the middle in this manner that won’t allow you also to eavesdrop on credentials sent in plaintext or MITM the entire thing.
> Every other scenario requires you to be able to MITM the traffic which is pretty much already gameover.
MITM attacks is one of the major reasons for using TLS/PKI. You might have total control over the TLS connections I make to a website, but you won't be able to inject information in either direction. All you can do is observe the amount of data being transferred and terminate/hang/throttle the connection. If you send something effectively different to what was intended by the sender, the recipient will detect it as a TLS error and terminate the connection.
This attack involves injecting information before STARTTLS which puts the connection into a state that the client is not aware of, this is not normally possible in plain TLS connections.
MITM attacks today are extremely difficult to execute. Most local networks are immune to traditional types of attacks such as arp spoofing/poisoning and wifi networks outside of one’s home have had client isolation enabled by default for at least a decade now.
So this leaves internet scale MITM which means TLAs and other actors at a similar scale.
The TLDR here is that if you can MITM a clear text protocol without any integrity verification then you can alter the message, I’m honestly not entirely sure why is this report worthy on its own.
The issues with STARTTLS are known since the RFC was published.
> MITM attacks today are extremely difficult to execute. Most local networks are immune to traditional types of attacks such as arp spoofing/poisoning and wifi networks outside of one’s home have had client isolation enabled by default for at least a decade now.
I find this hard to believe when authentication of WiFi itself is basically just SSID + PSK (ie, if you know both, you can either connect to an access point, or you can be an access point that other devices will connect to ("evil twin" attack)).
If there's a reason for MITM attacks not occurring nowadays, it's probably that they're usually not very useful because everything important is meant to be protected by TLS, not that they're difficult.
Your points seem to imply that TLS in general is useless, which I think you'll have a hard time convincing security-minded people of.
> MITM attacks today are extremely difficult to execute. Most local networks are immune to traditional types of attacks such as arp spoofing/poisoning and wifi networks outside of one’s home have had client isolation enabled by default for at least a decade now.
That's a strong claim. Most home networks most certainly aren't. There's also no enforcement so someone evil can always set up an evil twin, so what that the original has client isolation.
I would imagine ~99.9% of email submitted by end users is done over HTTPS which are highest risk. Servers talking to Servers are almost entirely wired so risk there is much lower which is probably why email providers have been slow to patch this.
Yes but then this attack doesn’t work either, you need a client that is happy with either unencrypted SMTP or one that will
attempt opportunistic encryption.
For awhile (2018ish - now), many large companies (Google, for example) were pushing everyone from STARTTLS and trying to move away from implied TLS. It was dumb then and of course it's still a bad idea.
Binary (or text) protocols needs to be de-bundled from transport layer for _a lot_ of reasons (looking at you QUIC).
Once we conquer this, we can de-bundle port number from service location.
Do you have a link for Google pushing people away from latent/implied TLS and towards STARTTLS? My recollection was the exact opposite (companies have increasingly pushing to kill off STARTTLS), but it's possible I've misremembered.
In the early internet years, your ip address was your identifier. It would have been absurd for your ip address to ever change. That changed quickly.
In the early internet years, fixed ports were the service locator. Yet this has not evolved.
As another commentor said, you can now only run one instance of a service per IP, which is aburdly wasteful of the possible ~65k inbound ports. TLS and SNI have come up with workarounds, but the root problem is still, port numbers should be available for anything, and we should have a system for service location. DNS is certainly an option, but there are other practical ways to do it.
Sadly it probably won't ever happen because of the numerous packet forwarding layers that have baked-in security expectations that "HTTP = 80" and "HTTPS = 443", etc. It'd still be nice to reach that promised land, but I think the damage has already been done.
I know it's not viable, but yet I've always felt IPv6 would have been a nice opportunity to extinguish port numbers. Just give each service its own IP from the /64 the host was allocated.
Then you could have one DNS entry per service. That would have been perfect for residential IPv6 with dynamic prefixes, which require dynamic DNS anyway.
Ah I get what you mean now. For the most part I just don't care about similar things running on the same port, because I've got many thousands of addresses I could just choose to listen to. I can have tons of different things listening on :80 or :443 or whatever on IPv6.
> which is aburdly wasteful of the possible ~65k inbound ports
What is being wasted? It's not as if those extra ports have additional bandwidth or processing power available to them that you wouldn't otherwise have. You also need a spare port if you're going to do any outbound connections.
Worst case, as far as I can see, is that CGNAT providers might require more outbound IP addresses than they might ideally need if more ports were actually used; but in the general case, there is no reason to care about "wasted" ports.
what is wasted?
IPs.
not like IP addresses themself were not an actifical limit - but let set aside this for now.
if i setup an MX, it still needs to be on port 25, because the vast majority of MTAs are not capable of service discovery; and there is no Host header or SNI/ECH for SMTP, so for each mail.DOMAIN.TLD i have to buy an IP, just to put them over a single multi-domain postfix/exim in order to please those MTAs who are requisiting the HELO name to match to the MX's reverse record.
how wonderful a world-wide adoption of rfc 6186 would be to able to put multi-domain mail server on a single IP.
Using port number as a service identifier sucks because it means you can only run e.g. one normally-accessible SSH server per IP address. One potential solution is to look up port numbers via DNS SRV records[1], but very few protocol standards demand clients do this.
HTTP and SMTP have their own ad-hoc, partial solutions to this (vhosting, MX records), but most every other service requires a unique IP address per instance.
We found more than 40 vulnerabilities in STARTTLS implementations.
In other words, the bugs are in faulty implementations, not specification.
Incidentally I was just reading about STARTTLS recently (i.e. within a month), and having to implement it, looked at the discussions about command injection, whereupon my first reaction was "seriously?"
This works by attaching additional commands to the TCP packet that sends the still unprotected STARTTLS command.
Am I the only one who thinks it should be bloody obvious that the first byte after the "\r\n" that ends the STARTTLS command should be the first byte of a TLS handshake, and that anyone who uses the term "TCP packet" needs to learn more about how TCP works?
Read bytes from the network into a buffer. At each \r\n (or \n to be more permissive), stop and parse the command. If it's STARTTLS, whatever is left in the buffer is the start of the handshake so feed that into the TLS layer. In my implementation, if I wanted to deliberately make it vulnerable, I'd have to make it more complex --- set a flag to remember, and do the TLS handshake only after parsing the whole buffer; but then, as everyone should know, TCP is a stream, so what if the buffer has a partial command in it? No. The only thing that makes sense is the bytes after STARTTLS\r\n are a TLS handshake, and maybe application data if the buffer is big enough.
Reading the linked Postfix article about how its design made it vulnerable confirms my suspicions --- they didn't understand TCP.
> In other words, the bugs are in faulty implementations, not specification.
Problems with STARTTLS have been going on for atleast a decade. At some point you need to start wondering if the design isn't the problem.
Besides, why would you even want STARTTLS? Even the authors of the RFC can't come up with a good reason to use STARTTLS [1]. Most of these reasons are from another era:
> Separate ports lead to a separate URL scheme which intrudes into the user interface in inappropriate ways.
Actually knowing in advance if something is going to be a secure connection (like HTTP vs HTTPS) sounds pretty desirable to me.
> Separate ports imply a model of either "secure" or "not secure." This can be misleading in a number of ways. First, the "secure" port may not in fact be acceptably secure as an export-crippled cipher suite might be in use. This can mislead users into a false sense of security. Second, the normal port might in fact be secured by using a SASL mechanism which includes a security layer.
Export ciphers are obviously not a thing in 2024 anymore. And "using a SASL mechanism which includes a security layer" where apparently someone made their own alternative to TLS sounds like a really bad idea.
> Use of separate ports for SSL has caused clients to implement only two security policies: use SSL or don't use SSL.
That actually sounds desirable.
> Port numbers are a limited resource. While they are not yet in short supply
Only very few protocols get an actual IANA assigned port number nowadays.
> PostgreSQL 17 adds a new connection parameter, sslnegotiation, which allows PostgreSQL to perform direct TLS handshakes when using ALPN, eliminating a network roundtrip. PostgreSQL is registered as postgresql in the ALPN directory.
I'm looking forward to being able to offload PostgreSQL TLS to a standard (non-pg-specific) proxy.
I was poking around recently. Adoption appears very low on a per-domain basis [1], but high-volume mail providers like Gmail support it.
Relatedly, DNSSEC adoption (and presumably DANE) is finally decreasing. So MTA-STS looks like the path forward. Lots of folks have been predicting this. [2]
It's quite a bit more accessible than DANE, so uptake has increased in the last few years. A bunch of high-volume operators support it, like gmail. But support also exists in newer MTA software like Stalwart, Maddy and ZoneMTA. (Though there are ways to add MTA-STS support to Postfix as well.)
MTA-STS has a great benefit that one doesn't need to deal with the antiquities of their registrars or TLD operators, just set up a few records and run a web server. WebPKI in general is also significantly better than DNSSEC. So you win twice.
Though MTA-STS doesn't cover the part between MUA and MSA unfortunately, I've seen an abandoned draft for MUA-STS but that's about it.
You can actually run MTA-STS on Cloudflare Workers so the web server piece isn't even necessary. It was fairly easy to implement and works well for me.
Server to server communication still relies on STARTTLS all the time, and very often simply ignore cert errors or, my favorite, attempts STARTTLS and fall back to non-encrypted if handshake fails due to host name mismatch.
the dumbest thing is if the server refused STARTTLS or the client doesn't send STARTTLS you are supposed to still continue if you strictly follow the standard
I really hope no implementation actual does support this behavior without setting some really dangerous sounding settings (but I'm pretty sure some probably do).
at least they did standardize directly connecting with TLS by now (it not "that" long ago that while supported in practice it wasn't technically standard complaint)
> you are supposed to still continue if you strictly follow the standard
Which standard? RFC 3207 (for STARTTLS over SMTP), 2002, says: "If the client receives the 454 response [TLS not available], the client must decide whether or not to continue the SMTP session".
I recall when I thought STARTTLS was a good idea...
Now, I know than having a transparent port and a TLS port is really important.
For many protocols, a transparent port is "enough" and crypto should actually be handled in a upper layer (like for email). But the transparent protocol should be "retarded", like for basic/core SMTP, S stands for 'S'imple.
A bad actor able to glean metadata from a transparent port can certainly do much more anyway.
As someone who self-hosted email servers, here's why it is a big deal: we all know that HTTPS is basically HTTP with implicit TLS, which is a common knowledge and industry standard of today.
But back in time when people still don't realize the importance of encryption, and most computers process crypto primitives in software, to the time when SIMD was not even a big thing (3DNow, MMX, SSE...) and only some high-tech niches and games such as Quake and Unreal Tournament demands it (where we obviously know today SIMD is super important for everything that can be processed in automatic vectorization with advancement in compiler optimization tech, including the fancy matrix/tensor math you run with your transformers and LLMs), which means they are slow because back in 30 years people don't have the luxury to enjoy widespread cryptography, only few nerds are adventurous enough to go deep into the abyss, and worse, still have to fight a war with the US government that they have military exports control on cryptography [1], since that's when RSA (the company, not just the algorithm) was prevailing, and everyone is still discovering new applications of cryptography to the commercial internet, one of which was smart cards, but that's not convincing enough to the US government cryptography should be set free from controls.
Today we even have accelerators for AES called AES-NI and the same for SHA too, and we even have those on embedded systems, MCUs such as Cortex-M or ESP32.
And so, cryptography is not only slow, not readily available and researched throughout, but also having legal troubles 30 years ago, so the predecessors used something called "opportunistic encryption", aka STARTTLS in your email client. All it means is that you start a TLS session on demand with a plaintext command (heck everything before STARTTLS is plaintext) indicating the following data would be new TLS handshake and starts the state machine exchange process.
Basically, they want to make sure that if the outcome of crypto wars turned out to be favoring the government, ala the three letter boys (FBI, CIA, NSA), we can still have some countries that could use TLS, hence the opportunity part, but this is obviously hell because everything before STARTTLS including the STARTTLS command itself is subjectable to MITM.
Today this is a thing of the past, including the crypto wars which it eventually sided to the people and flourished all e-commence businesses because now we all can enjoy safe and secure internet, and we all agreed that implicit TLS is the only way to go. I would argue that the success of TLS is part of the reasons internet exploded to an exponential scale because now banks can make secure transactions without someone sniffing it, and increased liquidity to the internet businesses, and that itself is a virtuous cycle.
And some people dare to have the audacity and have attempted to even remove implicit TLS from official IANA port list in the form of RFCs, only to be met with protest. This is why there are a lot of misconceptions online saying that implicit TLS is bad (those RFCs to remove implicit TLS were initially accepted but then quickly retracted), and often give confusing information which port people need to use. Basically, they are reinforcing the reminiscent knowledge 30 years ago.
Protect cryptography and thus your privacy and security at all costs.
If you want more technical details, we've put up a webpage back then, and published a background paper on those vulns: https://nostarttls.secvuln.info/