I am...not a fan of Golang, as I have made pretty clear around here on occasion. But I'll give credit where credit's due, and this is a good decision on the part of the people maintaining x/crypto/ssh. Not the tooling vendor's awful response--I'm pretty sure I know who it is, and if not there's two of 'em because I've had these conversations before--but the maintainers are doing the right thing. This probably shouldn't have gotten out the door without host key verification in the first place (and that ties back into the reasons why I do not like or trust Golang or its community when it comes to tools that I have to consume), but it's better to bite the bullet and fix this now instead of letting it fester.
(The "PGP is too hard for discussing security issues" thing, though, is total nonsense. Can't be doing that.)
Doesn't the writer call out hashicorp specifically by name in the article?
I used to respect this vendor and recommend their tools to others. I am having to rethink this a lot. I am no longer happy to recommend Hashicorp products to others.
Oh, ha, I glazed over that paragraph! But yeah, it was totally Hashicorp I was thinking about; this sort of reliability-thoroughly-optional thing shoots through a lot of their tools and keeps me pretty far away from any of the ones that might touch live environments. (Packer and Vagrant are fine.)
blog-post author here: all of them written in Golang which use SSH.
Packer, Terraform, Vault, all of them.
Using a bastion host in your configurations doesn't help if you invoke the tooling on your laptop, since the connection to the bastion host is done with the SSH package, again with no host-key verification.
Damn, I was planning on using Vault eventually. This attitude towards security gives me pause about put all my secrets into it. Is there any alternatives?
> I am bemused by an approach to accepting
> security reports which is to go through the
> motions of having PGP public keys available
> for people to use to report issues but upon
> receiving such a request ask for it to be
> submitted without PGP because digging out
> the keys is too much of a hassle.
I was the one who replied with that, asking him to resend without PGP, for two reasons:
1) both of our MTAs do STARTTLS. And gmail is only TLS.
2) 99% of the PGP-encrypted emails we get to security@golang.org are bogus security reports. Whereas "cleartext" security reports are only about 5-10% bogus. Getting a PGP-encrypted email to security@golang.org has basically become a reliable signal that the report is going to be bogus, so I stopped caring about spending the 5 minutes decrypting the damn thing (logging in to the key server to get the key, remembering how to use gpg). But I recognized him as a knowledgeable person from the Internet, and I knew (1), so I just asked him to send without PGP to save me 5 minutes.
Even if I'd used PGP, I would've just replied cleartext anyway to our security@golang.org list, except all the MIME would've been garbled and unreadable.
In summary, the PGP tooling sucks (especially in gmail, but really everywhere) and it's too often used by people who are more interested in using PGP than reporting valid security issues.
> I recognized him as a knowledgeable person from the Internet, and I knew (1), so I just asked him to send without PGP to save me 5 minutes.
I understand where you're coming from but this bit of logic I don't get. Especially if you recognized him, then clearly it's worth spending the five minutes.
The five minutes you personally save, someone else has to pay them. The person who originally sent you the email now has to read the response, process it, send you another non-encrypted email and/or enter into a meta-debate with you. They end up mentioning it on their blog and now we're all having this meta-meta-chat about it on hacker news, wasting several man-hours on it.
Negatively-anticipatory subjective analysis of the effort and difficulty of a task that needs to be done and which the brain has not "task-switched" into actually doing yet. (What's this canonically described as?)
Something we all do, and which I'm personally very guilty of.
In this case it was exacerbated by "99% of PGP-encrypted security reports being bogus" (to paraphrase).
I can clearly see both sides of this coin though (security!), and I'm not at all discounting what you're saying.
It's both ridiculous and horrendous that PGP remains an operational holy grail after how many decades?
STARTTLS is not required to be used (even if the peer advertises it), can be stripped from the advertisement by an MITM (and the relevant RFCs say you shouldn't configure your MTA to require STARTTLS for every peer), and the standards even say that the session should continue even if the certificate verification fails (absent any site-specific configuration).
This is strictly worse than the problem you're writing about (lack of SSH host key verification) because at least with SSH you know the link is going to be encrypted (perhaps only between you and an MITM, but I digress...), but with ESMTP you can't assume even that. For example, it is perfectly acceptable for Google to one day decide not to negotiate TLS with ESMTP MTAs it's delivering mail to. You could check the message Received: headers on all the E-mail you receive to see if it was used, but by that time it's too late.
Those figures are both good and depressing to know, thanks.
I figured the PGP usability problems were severe enough that I did not call you out by name in the post. This aspect was merely a darkly amusing aside leading into the main point of how the Golang devs handled this so well.
You didn't know the content of the report, so the irony is only clear in retrospect: the whole point of the report was about vulnerability to MitM attack and email MX->MX delivery is highly susceptible to that without some kind of trust anchoring in place, whether DANE or MTA-STS, neither or which is in place for the golang.org domain.
So the fact that TLS is _advertised_ by the gmail servers, which handle golang.org mail, doesn't mean that the advertisement reaches the sending mail-server.
I've since configured my mail-servers to always require verified TLS for outbound mail to the golang.org domain, as a manual override.
> 1) both of our MTAs do STARTTLS. And gmail is only TLS.
What you're implying here, i.e. that the communication is encrypted anyway, isn't how E-Mail works at all.
Even if the GMail MXs are configured to only accept E-Mails over TLS, the user's sensitive E-Mail may, depending on his settings, traverse multiple servers with less secure settings before a connection is ever initiated with the GMail MXs.
If someone's E-Mail is going to be MITM'd it's far more likely to happen between the Internet café he's sitting in and his ISP's badly configured SMTP server than between his ISP's SMTP server and GMail's TLS-using MTA.
GMail using TLS will do nothing to protect against the message being MITM'd, whereas GPG would, because it's end-to-end encryption, unlike TLS settings for individual MTAs in a possibly long chain of mixed or no encryption before a message reaches you.
Brad knows me slightly better than that and there was a bit more factoring into his decision. As I'm one of the maintainers and security/release people for Exim, a major MTA, I'd hope that the SMTP server on my personal domain is not too badly configured. :-D
You're apparently someone with at least a bit of influence at GOOG, which makes Gmail. If you cared about PGP, maybe you could use a bit of your power.
I'm confused about the use of "PGP-signed" vs. "encrypted" here. Why do you need to decrypt PGP-signed emails? I guess they were signed AND encrypted and the problem was the encryption, not the signature. So presumably just PGP-signed cleartext should cause no problem for security reports. Or is that problematic too for some gmail-related reason?
Advertising that you use pgp and then in practice not using it is kind of daft, but is there actually much benefit to using encryption to initially report vulnerabilities?
If we're talking serious vulnerabilities (e.g. can allow remote code execution) for a project used by a large number of people, then absolutely. It can be weeks between initial report and a fix being pushed depending on the vulnerability, and even longer for people to update. It's a pretty good niche: intercept emails to maintainers for important projects and either sell the vulnerabilities or use them yourself in that small window. It'll net you a pretty steady stream of 0days if you can pull it off and you choose the right target. Encrypting communication makes that strategy a lot more difficult, and more invasive.
It just seems to me that the window is small (and you have to write a weaponised exploit in that time too), and the only people likely to be able to reliably intercept the email are nation-state actors who do bulk email slurping, and they're probably sitting on a pile of zerodays anyway. But that's just gut feeling so it's likely wrong.
I'll admit it's not a likely scenario, it's just good trade craft.
The window isn't as small as you'd think. The hacker would have all the malware written except for the infiltration point. And a skilled team, who is familiar with the software you are targeting, can go from a whitepaper to something actionable in a couple weeks. Less if they don't sleep much. Even better, a lot of these vulnerability reports come with POCs that can be pretty easily adapted to your needs.
As for who could pull this off, you don't necessarily need nation-state resources. In fact, bulk email slurping probably wouldn't help, since STARTTLS is pretty ubiquitous. Since you're focusing on a single target, your best bet is compromising the SMTP server. If that's someone's personal server, that's well within the abilities of a moderately skilled group. If it's managed email (say gmail) it gets much harder, but maybe they get lucky and figure out your password, either from good guessing/brute force or from an account leak.
Of course, a group could get access to those vulnerability reports by hacking the laptop of someone with the keys, but at least using PGP lowers the attack surface.
This is probably a niche use; all the potential RCE vulnerabilities I've found in projects with large numbers of users have involved projects with no formal vulnerability reporting process at all, where just finding a private way of reporting the issue was a pain.
Using pgp is good for +30 cryptolluminati cool points. It also reduces your kibitz surface, which is useful for keeping the guardian from writing incorrect hit pieces about how you don't care about security.
> Mind, we are overdue for some nice convenient tooling to have a PGP-decrypting-and-reencrypt-to-subscribers mailing-list software tool for use for such public contact points.
I love Digital Ocean, but they do the same thing with their API. I wrote to them about it years ago, even talked to some developers there, and the general explanation is the same: Screw around with cloud-init to get the public key.
If you use the DO API to provision servers my feature request is here:
For hostkeys on DO you can probably get a script to run that'll request a signed certificate from a server you own. The signed cert can be validated fully by clients.
Or go with a convergence style system and probe it from multiple locations.
Or just give up and go with TOFU - if you never get an error even on different connections, you probably haven't been mitm'd.
While I really love the stable nature of Go and its standard library, I am happy that this breaking change was put out there in the interest of security.
This issue hit me while building a tool for internal use at my employer. I am using the glide vendoring manager for this project, added another dependency which triggered an update of all other dependencies. At that point my tool broke and forced me to actually think about host key verification.
> Security. A security issue in the specification or implementation may come to light whose resolution requires breaking compatibility. We reserve the right to address such security issues.
I didn't want to imply that. I meant that this kind of fix is a good reason to break something, and I am happy that they quickly reacted to this issue, and don't change everything all the time even though it is in the x/ packages and not covered by the standard library stability promise.
In general I am very happy that the big emphasis on a stable APIs was taken up by the community, and that we have a lot of stable packages out now (even though they might not be 100% stable like the standard library). Since I also have to work with NodeJs where changing APIs and packages are much more common, I came to really appreciate that fact about the ecosystem.
It's a special case though. The golang.org/x/ packages are experimental but also candidates for promotion to the standard library. Eg, "context".
But this sort of issue is exactly the sort of real-world review and hardening which justifies having a namespace for stuff to go _before_ it becomes stdlib.
I'm really impressed with the quick response from the golang team. The fact that they didn't mind introducing breaking changes shows that their priorities are right.
It's amazing how many people out there consider MITM as something they don't have to defend against. If you're a developer you have to assume your system will be MITMed. It doesn't matter if you're on the Internet or behind a firewall. Trust on first use is not a good solution because someone can tailor their attack against that first use.
I see so many people who just don't bother when using cloud VMs. They let the instance create a host key when it starts up, they have their own ~/.ssh/known_hosts symlinked to /dev/null, and they use -o "StrictHostKeyChecking no" with ssh.
What are some clear signs that this is happening to you? I track the url requested on my server(s) and some of them don't make sense/looking for exploits like wordpress login exploits. I don't know I have SSL/A+ according to Qualys, I'm kind of drawing a blank where MITM happens. I've seen/read about it before. Heartbleed? No I don't know... really tired, but interested, gots to Google.
edit: I'm still on LAMP stack for clarification. Too bad to hear about Golang though I'm still looking to learn it I hear a lot of great things about it.
What does that mean out-of-band? I'm not good with SSH, I'm not using 2-factor key based, also in general it makes sense to have separate servers right? Like one server that transfers request to a server closer to another country. Not Cloudflare but your own thing assuming you rented from different datacenters in the world. Sorry not related to the question.
edit: literally band? Like another wavelength/connection?
Out-of-band means you need to transfer (or at verify) the ssh host key over something other than ssh for the initial connection. The host key is how the client verifies the server is who it says it is. The client caches the key after the initial connection. But for that initial connection, there's no way to know that the key isn't that of a MITM attacker.
One way to get the key out of band might be from the AWS console for example. Presumably that connection is protected via HTTPS where the CA infrastructure (theoretically) can protect against MITM attacks.
BTW, it's also possible to setup ssh to use certificates instead of simple keys which might make sense depending upon how many hosts you manage.
Thanks a lot for the information. You mentioned AWS console, that's not service-specific right? I haven't used AWS before. At any rate, lots too look up/research I appreciate your time.
At the moment I'm just dealing with a cheap domain-mapped single-core vps
In that case, from an existing client with the SSH host key installed, or with your VPS provider's login shell system (if any), read the files '/etc/ssh/ssh_host_$ALGO_key.pub'. Each of these files will yield a line like this:
These are all different hostkeys that the server might give to clients, depending on their mutually-agreed signature algorithm. This is in the same format as a line in the known-hosts file; all you need to do is replace that last bit (in my example, root@ip-172-31-17-215) with the hostname by which you log in (e.g. someserver.example.com). Once you've done that, append the results to the known_hosts file, either for your user account (~/.ssh/known_hosts) or system-wide (/etc/ssh/ssh_known_hosts).
tl;dr: replacing HOSTNAME with the hostname by which you access the server, run the following command on your server and append its output to ~/.ssh/known_hosts on new clients:
Thanks a lot. Right now I just use PuTTY, possibly SSH by command line (OS terminal). I'm not sure what I would put for the hostname that you mentioned (some domain) I imagine this will come after you do the out-of-bandwidth access thing which at the moment I just have one server but I think you can do it on the same server "on another bandwidth" yeah. I'm still not 100% but thank you for this.
A different network connection (With physical infrastructure you might actually use a completely different network) that you can use to move private keys or any other sensetive information without exposing it to the Internet. We use such a network to monitor all of our infrastructure, and for iLo.
There are bots who mindlessly throwing HTTP requests on some known vulnerabilities, they don't really check whether you're running wordpress or not. Every web server bombarded by those requests, that's not something you should be worried about.
Yeah I don't use Wordpress but I just happened to see them looking at logs.
I am worried because I don't know. I covered some basics like SSL, XSS (htmlescape), sessions are redirect related, PDO/sql... no ddos/load balance... backup backup
That is what we like to call the background noise of the internet. Scanners are always scanning. Try this as an experiment sometime: make a brand new instance on your favorite cloud provider, and just run tcpdump on it for a few hours. You'll see all kinds of cool things: from people scanning for HTTP, Telnet, people scanning for SSH servers with weak passwords, router exploits with shellcode embedded in a UDP packet, back-scatter from DDoS attacks, cool stuff. Honestly looking at each packet and trying to figure out what's going on is a pretty great way to learn.
I did trace one of them and it lead to some "what appears to be" fake cloud provider. It had a cliche login form/company name but it didn't seem legit.
I'm not sure if it's possible to "bounce off" an attack from someone else's ip, by "attack" I mean a specific requested URL that is looking for an exploit.
If you want to study the background noise, I think you would have better luck reacting to it - as well as running tcpdump, set up a trivial HTTP and SSH server and observe the interactions.
I think you're being downvoted because there are two readings of your first question and reading the whole paragraph is needed to understand that the flippant reading "Psh, you say this is happening to you but I don't believe you, show me some evidence" is wrong and the humble reading "hey, how can I know that this is happening to me -- is it in the log files?" is correct. (It could also be that you're confessing ignorance even though you're also claiming certification though.)
The easiest MITM attacks for a civilian to perform involve a wireless router that traffic is routed over, with mildly more complicated but similar setups near a server -- say an untrusted server farm for example, or a malicious employee at a self-hosted location. More complicated attacks require messing with DNS so that people wanting to connect to that host instead connect to you; and really big entities like governments or advertising-hungry service providers could of course try to set something bigger up that affects more people.
The basic necessity is that the MITM computer needs to have an IP address, because the premise of MITM is, "you issue a request to connect securely to server IP 123.45.67.89, but you send your packets through me, and I have access to some IP 98.76.54.32. So now I intercept them rather than forwarding them on, and I connect securely to 123.45.67.89 myself, as well as to you. Now everything you try to send to them gets through, and its response gets back to you, but not before I intercept it and decrypt it and store it for later retrieval.
So the simplest way to do something like SSH is to imagine that I ask for your public key, you give it to me, I encrypt a shared private key with your public key, I send it to you, you decrypt it, and we use this private key for the rest of our communication. (That's not what SSH does, SSH does Diffie-Hellman, but similar reasoning applies.)
A MITM attack is then, you ask for the public key, I make a public/private key pair, I send you that public key, I ask for the remote server's private key for myself. You send me a private key, I send the remote server a different private key, and when you send something to the server, I now decrypt what you sent me, then encrypt it and send to the server.
Your main means of security in this scenario is actually 100% the same as your main means of security in SSH, it is that you trust that the private key of the server does not change very often. Therefore when you connect, you store the first public key you ever see; then each time you connect, you double-check it's the same or else you shout at the user "HEY! someone could be overhearing your data! Are you SURE you want to connect still?"
What's at issue in the article is that this crucial step was totally ignored by Hashicorp, it was brought to their attention and they just missed it.
The SSL thing yeah I tested my Apache configuration on Qualys it's an A+ I got help from people. It was a few weeks/month? ago... recently there was some new bug or something found and I was briefly dropped to like a C and then fixing one cypher suite or something made it go back up to an A+. I might have to deal with this again when I switch to Lets Encrypt.
I saw the wireless thing when briefly looking up MITM attacks.
>you issue a request to connect securely to server IP 123.45.67.89, but you send your packets through me, and I have access to some IP 98.76.54.32
Sounds like a VPN no? Haha (not a serious remark)
You say decrypt, I thought you can't decrypt or hard to if you're using SSL?
Yeah this is beyond me right now the whole time I'm thinking Port 80, regular web not SSH (22? or whatever port assigned), it's hard to think that's intercepted as well but why not, its just a port? Yeah thanks for taking the time to write this, I'll refer to it when learning more about this "have my bases covered"
I'm always skeptical connecting to public networks (try not to) or use a burner device or something without typing in passwords/https.
You're not wrong to notice the VPN side: what are most of our examples? Malicious service providers! What's a VPN? An intermediate service provider!
I did say decrypt, and I did really mean it. SSH and TLS are not nebulously "hard to decrypt", otherwise we wouldn't use it because our servers couldn't decrypt the communications from our clients. They are hard to decrypt if you don't know a shared key. In MITM there are two shared keys: the client shares one key with the interceptor, the interceptor shares another key with the server. That's why the interceptor can decrypt.
Furthermore if we're talking the normal open Internet, the interceptor-server negotiation is dead simple because it looks to the server like any client-server negotiation. If you let anyone on the Internet access your pages through TLS, there is no hope to catch MITM on the server-side. It has to be defended client-side.
There is fundamentally no way to stop an interceptor who you have to send messages through from throwing away those messages and sending you the reply, "why yes, I really am https://mail.google.com/, here's my public key, let's negotiate a shared key so that we can talk in secret." Nothing. Because, you don't have an encrypted connection before you've negotiated that shared key. You aren't encrypted before you're encrypted. So for that setup phase, there's no way to stop them from throwing out those packets and sending that reply; we can only hope to detect that they are lying.
Now they are only telling us one thing, "here is my public key", and we want to know if they are lying about that. We have come up with exactly two strategies to deal with this client-interceptor side.
1. Like SSH does, admit defeat: but only for the very first connection. The very first connection, yeah, that could be intercepted. Then I will store your public key forever. Every time afterwards I will either use that public key, or else I will fire up the klaxons warning my user, and they will have to manually approve connecting and storing the new key. And that means you'll have to keep intercepting my traffic--even if I take my laptop to a different location or whatever--or else I'll find you out eventually.
2. Like TLS does, communicate some way of authenticating that public key through some other mechanism than the current internet connection. This usually happens quietly as part of downloading a web browser: that browser comes with a list of public keys of authorities whose digital signatures are trusted by the browser for authenticating public keys. If a site just sends me their public key? Forget it! They have to send me their public key plus a certificate issued from one of these authorities, saying "Yes, that is a valid public key for that domain," signed by a private key whose public key I know and trust. As long as I can verify digital signatures and they cannot be copied, I am good.
These are called "Trust on First Use" and, well, I'm not sure what you'd call what TLS does. Something like an "out-of-band" configuration of a "web of trust" I suppose. So those are the only options we've gotten working at scale, TOFU and OOBWOT.
Thanks I will have to go down this road at some point. Have to learn a bunch still. Sorry I didn't see this sooner, had a wall of text to look through (not yours other replies from other posts).
Golang seems to follow the 80/20 rule from the outset (or perhaps an even smaller proportion), which is perfectly fine. Some other languages' standardlibs try to offer a complete treatment of a particular problemspace from the start which is tricky to get right on first attempt. Those are the instances where developers complain about complex APIs, uneven abstractions, or the like.
However, one of the artifacts of a popular language having a lean-and-mean standard library is that custom code proliferates, and the Go community's distaste for frameworks (as opposed to libraries) means that the it's not just the business-specific edges of the code that's unique in each implementation (as you'd expect), but also a good amount of the plumbing and domain-specific control code and their immediate callers. In some other languages, where there's more of a culture for using a dependency to intentionally simplify your problem space in exchange for ceding control, this style would be derided as NIH.
The vendor's response here is a function of not only the vendor's own rationale and priorities, but also of the above developer philosophy. This is surprising to me, given that Go is an opinionated language, and yet opinionated third-party code driving your logic is frequently discouraged by its community.
On the other hand, the language maintainers' response was measured, proper, and commendable. They made a breaking change to an experimental API, and improved their product in the process.
Telnet does not support port forwarding, file transfer, host key authentication (trusted network ≠ trusted clients), etc., and is not installed by default on most systems. rsh only supports file transfer out of those, and is dead enough that it's not even in Homebrew.
Nor is there any reason not to use SSH just because you trust your network, except possibly performance of huge file transfers.
Well, I'd rather run something I know has limited security (eg rsh+/etc/hosts-allow over a closed vpn) than ssh without key verification - because then you throw out trust, and encryption without trust buys you very little. Or kerberized rsh/telnet.
As for port forwarding, if you trust the network, presumably you don't need forwarding? And for file transfer just use zmodem, or run rsync/ftp? (remember: you trust the network..).
I will admit that there's a minor convenience to be able to use one client/api/interface - but I'm not sure it's worth the tradeoff of suddenly not knowing if you should be trusting ssh to be actually secure, rather than just convenient.
Clearly the maintainers of the go package feel the same way (after some gentle prodding).
Hello! As the blog post clearly states, the vendor is HashiCorp. As the founder of HashiCorp and someone who participated in the initial report we received on this topic, I'd like to state our point of view from my own mouth.
I'd first like to be up front about exactly which of our software doesn't perform host key verification, since we have a lot of software and this CVE doesn't apply to most. There are three places that were identified as affected: Packer and Terraform with SSH provisioners, which both create a machine resource and can perform SSH connections to setup the machine; and Vault’s SSH backend in Dynamic Key mode performs SSH connections from the Vault server to hosts (other modes do not).
Any other usage of our software is unaffected.
We’ll discuss each of these cases in detail, since the details matter to understand our thought process and response.
Vault:
The SSH secret backend has three modes that can be used for generating SSH credentials: certificates, one-time passwords, and dynamic keys. Only the dynamic key mode ever actually makes connections to other machines, but more importantly, our documentation has always recommended that the dynamic key mode only be used as a last resort because of its various (documented) drawbacks compared to the other modes. With the addition of the ability to generate SSH certificates (which was on our roadmap for a long time and added in 0.7, prior to both the original report and the blog post), we did not explicitly mark the dynamic key mode as deprecated in our documentation, but we probably should do so.
Given that it is not recommended for usage (but maintained for backwards compatibility), we chose to warn users of this additional drawback of the dynamic key method, and documented the lack of host key verification (https://github.com/hashicorp/vault/commit/251da1bcdc27678fea...). As we stated in our response to the reporter, "It isn’t something we want to hide (and we’re not trying to) and we will document this."
Terraform/Packer:
Terraform and Packer support the ability to use "provisioners" to bootstrap a machine. In both, the provisioner is run very shortly after the machine is initially created, representing an extremely small window of attack. Neither support connecting to a pre-existing machine via SSH under normal use cases (you can make it happen through some advanced configuration trickery with Terraform, but it's abnormal). Because of this, we didn't register this as a high-priority issue.
However, we admit that this can be improved and we likely should've been more reactionary in our response. I apologize for that. We have added plans to improve this to our roadmap, covered in a couple paragraphs.
As the blog post states, the reporter suggested parsing console logs to determine the host key. And, as the blog post correctly says, we don't want to do this. There is a combinatorial explosion of complexity in supporting this, we have experience with this (due to Vagrant supporting this type of behavior), and we've found maintenance of this sort of functionality to be difficult to support over time. We came to this conclusion though only because there is a viable alternative: SSH certificate authentication. If a viable alternative didn't exist, we may have been forced to take the more complex route.
SSH certificate authentication was introduced many years ago and is broadly supported. This type of auth also provides authenticity to a first-use connection. We mentioned in our response email that this is something we're open to doing instead. I admit that in our response to the reporter, we explicitly said this "is not a priority" but shortly after decided to schedule this work for the next major TF release. We should've followed up again, but didn't.
And that's where we're at currently! I hope this helps make our response to the report and our future roadmap around this issue more clear.
Why do you think there is "an extremely small window of attack?" The mitmproxy is likely between you and the internet somehow, meaning ANY ssh connection will be MITMed. It is not like the MITM is waiting for you to spawn a machine before it pops into existence.
For terraform and packer aws providers you could use "get-console-output" since the host key is generated during launch and the fingerprint is written to the console
I really like your style, that you communicate directly and in detail to issues that come up. Do you also record these posts with the software for others to find in the future - who don't follow [HN] or twitter? If this blog goes away independently of the software, this discussion goes away too.
> Neither support connecting to a pre-existing machine via SSH under normal use cases (you can make it happen through some advanced configuration trickery with Terraform, but it's abnormal).
Is a `null_resource` with an s `file` or `remote_exec` provisioner really "advanced configuration trickery"? I don't do this often in production, but when hacking on a module it's nice to be able to re-provision a resource without going through the entire destroy/create process over and over.
Well. I haven't really started to learn golang yet. But sure that this breaking change indeed convinced me to do it. I have implemented an automated ssh session in another language there was absolutely no host key checking or tofu implementation even worse that they designed the api not to allow that manually. That was frustrating. But obviously the golang language designers and the entry owner and myself sharing the same concerns obviously. Thanks for writing about this
Really wonderful and thorough report. I'm not at all a security expert but manage to learn quite a lot from reading this post. Kudos to the author for giving context and background on the issues. If more security reports are written like this, the whole industry would benefit greatly.
1. how can an experimental library (x/) get a CVE?
2. what is "hostkey verification"? Probably the fingerprint check you usually get when you ssh into a machine + the blocking warning you get when the fingerprint of the machine suddenly changes.
3. if this is what "hostkey verification" is. How is it so hard to implement? create some sort of fingerprint out of the server's public key; prompt the user for input; cache the result.
Every host has a public/private keypair that it stores locally. The client stores the last confirmed (or manually added) host key in ~/.ssh/known_hosts. Host key verification is checking the host key against this list.
A lot of people ignore the importance of this part. It is also a bootstrapping issue: At some point you need to get the key into known_hosts. Most people do this by confirming it the first time they connect. That's usually fine for new servers, but what about old servers, e.g. if you're a dev who just joined the company? Most companies don't seem to have a secure way of seeding clients with the initial host keys. I've never seen it.
We solve this with a script that gets them from Puppet (via PuppetDB, its repository of host "facts"). That script has to hard-code the host key of the Puppet host, of course.
Another solution is using the X.509-style certs to sign each host with your own CA, but I haven't looked into that workflow. Edit: It's actually a much nicer workflow. You can sign users, too.
That's usually fine for new servers, but what about old servers, e.g. if you're a dev who just joined the company? Most companies don't seem to have a secure way of seeding clients with the initial host keys. I've never seen it.
One way to do this is via SSHFP (ssh fingerprint) records in DNS. Unfortunately they do not seem to be particularly widely supported nor widely used.
The "x" in golang.org/x/* does not mean "experimental". It also doesn't mean "extension". It by definition has no definition.
We thought that was a good idea for a while, but we've recently decided we should figure out what the rules are for the x/* packages. That is https://github.com/golang/go/issues/17244.
I'd expect an experimental library to possibly be filled with bugs and constantly evolve (in a breaking way) its API while having a "USE AT YOUR OWN RISK" message. Although that might not be how the namespace golang.org/x/ is defined.
CVEs can be thought of more as a public security advisory. Many vendors and distributions follow the CVE list. If there was no CVE, this issue might have gone unrecognised in your package manager or paid software product for a long time.
I'd hope these don't use experimental libraries. Although as others pointed out to me, this is not an experimental library :) (which was the answer I was looking for)
The golang.net/x/ libraries are widely used in production, so it's absolutely reasonable to use CVEs.
The word "experimental", in this context, means "not covered by Go's standard library's (rather strict) backwards-compatibility promise" (as can be observed in this case).
x doesn't mean experimental here. I think it's more like the x in javax, meaning extension or something. The compatibility document [1] isn't that illuminating:
"Code in sub-repositories of the main go tree, such as golang.org/x/net, may be developed under looser compatibility requirements."
blog-post author here, others are addressing the other points, but I wanted to answer Q1:
CVEs are for tracking response to issues, so that people can clearly communicate about what they are reacting to. Everyone using the library needs to update their code (unless they were already setting the callback) and so having a CVE lets them describe exactly what they're reacting to.
FYI, you can also instruct cloud-init to use a particular key pair by supplying it in instance metadata/user-data. This avoids the need for hacky scripts extracting public keys from console output (which may also be delayed by a few minutes after the instance starts).
When we built the new set of drivers for Neo4j, we decided to allow three modes: No encryption, Trust on First Use and Trusted Signature - there's no way to establish an connection without trust.
This was a terrifying decision, because of ease of use concerns. Having done so and shipped it, TL;DR: It worked awesome, outside of some early kinks in TOFU that we worked out - and now everyone can sleep well knowing there's not a single install that thinks they are running an encrypted setup when they really aren't.
Anyone that came back asking for a flag to disable host key verification seemed happy with our argument for why that's not really much different from just disabling encryption.
(The "PGP is too hard for discussing security issues" thing, though, is total nonsense. Can't be doing that.)