This shouldn't be downvoted. Docker has been very openly history towards Red Hat in the past. To the point of openly mocking their developers at DockerCon.
Hi, I'm the founder of Docker. It's not my place to say whether comments should be downvoted or not, and I don't want to ignite teenage-drama arguments over who was mean to whom at recess - we get enough of that level of discourse on the US political scene these days.
But I think there is an interesting topic to address here, that we deal with a lot at Docker.
The problem in a nutshell: if you choose to take the moral high ground and only promote your products with positive marketing (and that is our strategy at Docker - you will never see any Docker marketing content criticizing competitors), you are vulnerable to bullying and unfair criticism by competitors who don't mind hitting under the belt. Then the question is: do you allow yourself to respond and set the record straight? Or would that just legitimize the criticism by bringing more attention to it? On the other hand, not responding is also risky because it emboldens the trolls to take more and more liberties with facts and ethics. This dilemma becomes more and more pressing as you become more successful and more incumbents start considering you a possible threat to their business. Some of these incumbents have been defending their turf for decades by perfecting negative messaging. Like one competing executive once told me - "we eat startups like yours for breakfast". This situation can be bad for morale also, because your team sees their work and reputation dragged in the mud, and can interpret their employer's silence as a failure to stand up and defend them.
The most perverse variation of this problem is when trolls start preemptively painting you as bullies. If that narrative sticks, then you're in trouble, because any attempt to set the record straight will be interpreted as hostility. Now you have two problems: defending yourself against the bullies AND defending yourself against unfair accusations of being a bully.
The root cause of the problem, I think, is the diminishing importance of facts and critical reasoning in the tech community. We are all guilty of this: when was the last time you repeated a factoid about "X doesn't scale", "Y isn't secure", "I heard Z is really evil" without fact-checking it yourself? Be honest. Because of this collective failure to do our own thinking and researching, bullies have a huge first-mover advantage.
I see an direct parallel between the problem of corporate bullying in tech and the problem of partisan bullying in politics. And I think in both cases, there is a big unresolved problem: how do you succeed and do the right thing? How do we collectively change the rules of the game to make bullying and negative communication a less attractive strategy?
I tried really hard to make this a constructive post about a topic I care about. If you interpret any of this as hostile or defensive, that is not at all the intention.
When the top comment is a claim that we at Red Hat post incorrect information and that we at Red Hat are expected to delete said supposedly incorrect information without any technical explanation on why said information is incorrect, I do wonder who is the bully in your opinion.
The post is in fact incorrect. The reason Nathan is not sharing more technical details is to protect the security of Red Hat users.
Also, if hypothetically the full details made Red Hat look bad, is it fair to assume you would be calling Nathan hostile for sharing them? In that scenario is there any course of action we could follow that would satisfy you?
The article is about how SELinux helps in mitigating or even blocking paths that would lead to a working exploit.
The article explicitly states the CVE number and the fact that updated packages are available.
The article IMHO doesn't attack nor provoke Docker and its people. Yet the first comment posted here DOES contain direct accusations against Red Hat. I don't think that's helpful nor needed. That's all.
I still think that SELinux and Docker are a good combination and this article helps in understanding why.
The title of the article is "Docker 0-Day Stopped Cold by SELinux." The title strongly implies that SELinux would have prevented the issue in the CVE even without the fixes Docker provides.
Then the text of the article states:
"This CVE reports that if you exec'd into a running container, the processes inside of the container could attack the process that just entered the container. If this process had open file descriptors, the processes inside of the container could ptrace the new process and gain access to those file descriptors and read/write them, even potentially get access to the host network, or execute commands on the host. ... It could do that, if you aren’t using SELinux in enforcing mode."
So, not only does the title make this suggestion, but the text of the article downright says it.
If the claim is wrong, then Docker's security team is right to correct it. However, I think they should do so in a forum other than in the comments of a HN post, be thorough in their explanation, and maintain a professional, polished tone in any communications.
And, of course, Red Hat should correct and/or clarify the post as well.
Work with RedHat and let them issue a correction; HN seems like hardly the forum to be calling this out in such a manner. The fact that you even made the "bully" post leads me to believe that Docker views RedHat as a threat and their post promoting SELinux as an affront to Docker. I'd wager it didn't appear that way to the many individuals that understand both the benefits of systems like SELinux and AppArmour, as well as the difficulty in promoting them. Very interesting at the least; one could read a lot into that.
Honestly, Docker stepped in it here. This appears to be a theme, and with your posts in particular. I personally can't think of another project that causes more drama on HN.
SELinux is great, and it somewhat mitigates this CVE, which is what defence in depth is indeed supposed to do. There is a big difference between "Docker 0-day stopped cold by SELinux" and "Runc CVE mitigated by SELinux" which is what a factual headline would be, as it is not a 0-day, and affects the majority of container runtimes (it was derived from an lxc CVE originally). You should always use defence in depth to make exploits harder, but you should always have the humility to understand that security is hard, and is an ongoing task to make every part of the system more secure, while keeping is usable.
This drama is a bid... odd in tone to me. It feels as if theres a huge confrontation looming, where right now all Docker came out and said was "SE Linux only barely helped us, our software was actually so terrible that even with the best effort of SE Linux, it still was vulnerable".
This makes Docker look worse.
The only way SE Linux can look bad is if you were on the fence about its efficacy, and are only now hearing it can't even stop Docker's problems.
"A number of developers from RedHat were once very involved in the project. However, these developers had a very arrogant attitude towards Docker: They wanted docker changed so that it would follow their design ideas for RHEL and systemd. They often made pull requests with poorly written and undocumented code and then they became very agressive when those pull requests were not accepted, saying "we need this change we will ship a patched version of Docker and cause you problems on RHEL if you don't make this change in master." They were arogant and agressive, when at the same time, they had the choice of working with the Docker developers and writting quality code that could actually be merged. Another thing they often said was along the lines of "systemd is THE ONE AND ONLY INIT SYSTEM so why do you care about writing code that might be portable?" Or "even though the universal method works with systemd now. A redesign has already been planned in systemd, so do it the new systemd way and don't be portable."(This is in response to the fact that Docker writes to cgroups, and systemd would like to be the "sole writter to cgroups" some time in the future.) I think everyone got fed up with those people, and Docker has rightly pushed them out."
I think this is a pretty poor understanding of the vulnerability. Yes, runc was split out from Docker but it now is maintained by many companies, including Red Hat - so to suggest that Docker software was "so terrible" doesn't sit right.
Not to mention that this is a fairly gnarly CVE - a great catch by SUSE and Docker - and claiming the software is terrible because it contained this seems like a real stretch.
I'm not saying Docker is bad, only that having a Docker engineer come out and say 'no our bug wasn't fixed yet' shouldn't be the start of a confrontation.
It's not infrequent to see occasional tension between vendors around the wording and timing of security disclosures. There are frequent examples of this with Google's Project Zero for example.
As far as disagreements on the best way to handle a security disclosure, this one is pretty straightforward and was entirely avoidable. The vulnerability in question is a runC vulnerability, it affects equally all products with a dependency on runC (not just Docker). The vulnerability had already been patched in runC, and an update to Docker had already been released and announced. So it was not a zero day. A few vendors (not just Red Hat) have incorrectly announced to their users that they didn't need to upgrade to the latest version of Docker because their enterprise-grade commercial platform would "stop the vulnerability cold". In the case of Red Hat, the commercial differentiator is what they call "security-enhanced Linux" using SELinux.
These vendors are under a lot of pressure to justify the high cost of their enterprise subscription by demonstrating concrete value. A great way to do that is to describe a scary vulnerability in a well-known product like Docker, and show that buying their product is the best protection against it. That is why the article talks about a "Docker vulnerability" instead of a "runC vulnerability" - Docker is a better-known product so the story will be more impactful that way. And it's also why the vulnerability is qualified as a "zero-day" even though it wasn't: it makes the vulnerability scarier.
Red Hat was privately contacted to inform them of their mistake. They privately acknowledged the mistake. When the article hit the Hacker News front page, they were again privately informed of that as well. In spite of multiple requests, after several days they have still not corrected the article. This puts the security of Red Hat users at risk by continuing to tell them that an upgrade to Docker 1.12.6 is not necessary. This is especially disappointing because of the obvious conflict of interest. It was a perfect opportunity for Red Hat to set the bar high for themselves and remove any doubts that they might put the security of their users before commercial interest.
The saddest part is that RHEL and SELinux have genuine security benefits that could be explained in very compelling ways without these shady marketing tactics.
BTW, can you confirm than SELinux in enforcing mode really prevents exploiting of this runC vulnerability? Therefore, the argue on the post's correctness considers only RadHat's marketing war.
Because if the answer is "No", and there's some other way to bypass SELinux and exploit this bug, it raises more grave accusation of RedHad - false statement about the vulnerability workaround.
Thank you for clarification of your point. It really shows perfect example of the Red Hat marketing.
Can you please give a link to the announce from Red Hat or someone else urging their users that they don't need to upgrade? It would be the last thing closing the question.
The blog post being discussed here is the latest example. NOTE: the blog post has since been updated without acknowledging the inaccuracies in the earlier version.
$ wdiff -n -3 first latest
======================================================================
[-Docker 0-Day Stopped Cold by-] SELinux
======================================================================
SELinux {+Mitigates docker exec Vulnerability+}
======================================================================
Fixed packages [-have been-] {+are being+} prepared and shipped for RHEL
======================================================================
[-Centos.-] {+CentOS.+}
======================================================================
[-Stopping 0-Days with-] SELinux
======================================================================
SELinux {+Reduces Vulnerability+}
======================================================================
[-How about a more visually enticing demo? Check out this animation:-]
======================================================================
we were glad to see that our customers were [-safe-] {+safer+} if running containers with setenforce 1
======================================================================
{+Even with SELinux in enforcement, select information could be leaked, so it is recommended that users patch to fully remediate the issue.+}
{++}
{+This post has been updated to better reflect SELinux’s impact on the Docker exec vulnerability and the changing threat landscape facing Linux containers.+}
======================================================================
I'm not sure that first post's version can be considered as recommendation to not upgrade. It just shows how RedHat people was happy to see that bug was prevented by another subsystem. Me, as a sysadmin, would be happy to to know that I'm not obligated to upgrade urgently everything I have. For most sysadmins it can be considered as a workaround, already engaged.
You as a Docker developer see the post as an attack on your project. But most of sysadmins and kernel developers see it as a nice example of the fruits of invisible long work - when well cared system with accurately configured security restrictions saves from some vulnerabilities.
Anyway, it not means underestimation of the Docker and you great job. Sorry you've got stressed by all this noise.
Don't know if it was marketing material when you published it in that whitepaper, but it definitely became marketing material when the @Docker twitter account tweeted it (https://twitter.com/docker/status/768232653665558528).
I don't think the material you're referring to qualifies as criticizing competitors at all.
- The comparison table is part if an independent study, not authored or commissioned by Docker.
- The table shows the strengths and weaknesses of different container runtimes; weaknesses are highlighted for all of them, including Docker
- The table is used in Docker material to illustrate the point that independent security researchers consider Docker secure. Nowhere do we make the point that other products are insecure. I encourage you to read the whole material and decide for yourself.
- The context for this material was to respond to a massive communication campaign painting Docker as insecure.
Even when written, the study made inaccurate claims about rkt (the SELinux support in rkt is identical to that in Docker, because rkt uses the same code from libcontainer - there is literally no difference there). It certainly wasn't an accurate depiction of rkt's state of security as of August.
(Disclaimer: I implemented some, but definitely not all, of those security features in rkt, and I currently work at CoreOS)
Come on. This comes across as entirely disingenuous.
"This was independent, not authored or commissioned by us." - "the fact that we posted it on our Twitter as "why your containers are safer with Docker", posted a lengthy blog article in no way means we were criticizing competitors..."
"Nowhere do we make the point that other products are insecure." No, just "less secure".
"The context we were in was responding to a massive communication campaign ... " - so you were responding to criticism by what, exactly? Oh, yeah, that independent study that you just decided to post about?
Haters gonna hate, man. It's fine to correct misinformation, but in the long run -- and whether they say so or not -- there's much greater respect earned in taking the high ground and not dragging oneself down into the muck of name-calling, ascribing malicious intent to others, and other ill behaviors.
If I were your counselor, I'd advise you to do nothing other than stick to the facts; make the best product you can; delight your customers; take pride in the great work you do; and apologize openly and honestly when you make avoidable mistakes. You can't make everyone happy, so focus on the people you can, and aim to exceed their expectations.
>Haters gonna hate, man. It's fine to correct misinformation, but in the long run -- and whether they say so or not -- there's much greater respect earned in taking the high ground and not dragging oneself down into the muck of name-calling, ascribing malicious intent to others, and other ill behaviors.
This is a naive perspective that doesn't bear out in the real world. It's important to know that by taking the "high road", you are putting yourself at a competitive disadvantage. Someday, those with fewer scruples may have to pay the piper and their dubiously-maintained prosperity may disintegrate ... but then again, maybe not.
Most often, the truth is that large companies are pretty ruthless, and have consolidated such a huge amount of control that it's extremely difficult to do anything about anything they do or have done. They control the messaging, they have a reputation that supersedes any complaint an individual may make, etc. Those companies do slowly atrophy, but usually it's more because they've lost sight of the founder's vision that originally connected with the masses than that they're engaging in questionable tactics.
If you're taking a position out of principle, that has to suffice for itself, because it probably will cost you in material terms.
That's a funny comment, because you are "taking liberties with facts" exactly as lamented by shykes above.
You start with a grain of truth — something that actually happened in reality. In this case, it was a joke protesting systemd hegemony.
Perhaps you thought that joke was in poor taste. But lets leave that aside for now.
So you start with an actual fact. Then, you exaggerate/falsify it, changing the details pretty wildly, and present this story of something that supposedly happened. In fact, nothing like that happened... but it sort of feels like something that might have happened. It vaguely resembles the actual event that did happen (in that, a Docker employee did wear a badge with an opinionated phrase on it at a conference).
The key thing, though, is that what you describe is completely and utterly different from the thing that actually happened in reality world[1].
You might not even be the person who changed the details to make the story more compelling (and false). Maybe you got this information from a post shared on Facebook, or from an email forwarded by your uncle.
Either way, though, the impact of your comment is to pollute the body of discussion and degrade the collective understanding of this topic. (If this process feels familiar, it's because it is exactly the process that eventually caused the failure of the American democracy... just at a much smaller scale).
Personally I don't have any stake in the Docker/RedHat relationship and I don't care about it. I only looked up what actually happened[1] because the idea of a Docker employee wearing an official badge that says "I reject red hat patches" seemed so unlikely to have occurred that it sent my bullshitometer into the red.
Suggestion: when something smells like bullshit, don't eat it without conducting a bit of research.
That badge seems pretty unprofessional to me. I would discipline my employees for that sort of behavior, especially if it occurred at my own conference.
Meh...I wouldn't. It's definitely an ingroup-humor signaling thing, but I find it hard to believe that someone would read that and seriously get offended unless they're being self-righteous.
