Sounds like SolarWinds hired a good PR agency with the goal to deflect blame and make it sound like a big conspiracy. JetBrains being a Czech/Russian company makes it the perfect scape goat. As was pointed out, "solarwinds123" hints at very bad security practices which makes some unpatched system or weak password the much likelier scenario. It might well be that the intruders then manipulated the SolarWinds TeamCity config. Would have done the same with Jenkins...
And given that the malware version was signed using SolarWinds' keys, it's likely the attackers just built and deployed their compromised versions using the regular TeamCity CI servers. No need to exfiltrate the signing keys if you can get control of the CI/CD system. It's less likely to be logged, it has the same effect, and such servers regularly contact the outside world to download dependencies. And if the admin password to the CI/CD was "Solarwinds456!" or similar (as seems in character, if possibly overestimating their intelligence by adding "special characters") it wouldn't matter what CI/CD software they'd used.
On a semirelated note, what's with all the articles claiming the attack represents an extreme level of sophistication?
It sounds like the execution was skilled, but I haven't heard anything yet that seems technically novel or extraordinary.
Maybe I haven't read the right articles, but it sounds like solarwinds got pwned, and all these big targets loaded the malware onto their own networks...
Again, skillful execution, but it's not like they cracked an encryption algorithm or even did known-but-still-awesome exploits like rowhammer/spectre
There are 2 separate things going on, which often get conflated:
1. The actual malicious payload that was installed on the compromised networks was very sophisticated. It was discussed in the HN post when the news first went live, but the payload went to great and pretty ingenious lengths to hide its tracks and prevent detection. I remember a bunch of HN comments along the lines of "Wow, how did the companies ever even find that?!"
2. How SolarWinds was originally breached hasn't been released, and while the "solarwinds123" password is an unrelated issue, it does point to a pretty shocking lack of security culture at SolarWinds.
I'm going to put my chips down on the original breach being quite simple and mundane, but SolarWinds will go (and has already gone) to great lengths to emphasize the sophistication of the attackers to try to deflect blame.
If Amazon's research is correct that 250 companies were targeted, that indicates just as a matter of a scale there had to be significant resources behind the effort.
I'd call that sophisticated even if just in terms of organizational efforts. Additionally, doing this and not getting caught for 6 months or whatever takes significant discipline from every single person involved. You don't see that level of size, discipline and organization outside a handful of top companies.
I haven't been following this story very closely. Was each and every organization individually targeted after the SolarWinds compromise was in place, or was it sort of opportunistic and automated, where the SolarWinds compromise phoned home and most of the 250 orgs just got automated/scripted escalation and exfiltration, and only a few orgs got personal attention?
Something like 18,000 orgs got compromised IIRC, but the 250 are the ones where they chose to actually activate secondary payloads. So 250 got personal attention.
Most likely attack on open port, then they sent automated some social engereening stuff, and while looking at it later they've found some fish to fry.
Sorry for my English here, too complicated times from Finnish speaker.
I wouldn't be really shocked if this would turn out an Equifax-style attack (company negligence due to outdated software and lax security policies resulting to big advantages) combined with supply-chain attack (for customers of SolarWinds).
Exactly. As a developer that has used the CI&D infrastructure to finagle some things I didn't quite have permission to do, it doesn't surprise me that CI&D is a good way to escalate permission after the initial hack.
It is non-trivial to write secure deployment scripts/configs and ensure that access keys and credentials can't be leaked to anyone with dev access.
Reading between the lines, does this mean the attack may have simply been an on-prem install that was compromised, rather than every TeamCity install ever, or JetBrains' official SaaS version?
Any developer tools company worth their salt should be dogfooding their own build system, so they likely build IDEs with this tool. In the worst case, IntelliJ/GoLand/etc could be compromised as a result. This would be unlikely to mean there's malicious source code floating around, but it could mean lots of privileged access to software companies' networks. If the attacks are as targeted as the NY Times article makes it out to be, discovering the full extent of the damage may take quite some time ...
Unless you are also verifying that the signed commits are from particular users, and don't allow any tooling to create unsigned commits, that might work, but many of the issues we found weren't even about the code itself.
Start by limiting access to secrets within various pipelines.
There's no reason why a merge request on a small tool repository needs access to the AWS keys for production for example.
Splitting instances into different needs. The CI/CD pipeline used for deployment shouldn't be the same one that runs tests for developers.
Especially if that CI/CD pipeline allows for the docker socket to be mounted inside of the CI/CD pipeline so that it can spin up more docker containers (as an example).
It's all about limiting scope. Far too many of these systems are not appropriately configured setup, or are exposed with far too many secrets as environment variables and the like.
Where possible also provide time limited scoped tokens. So a deployment takes ~10 minutes? The AWS creds for that deployment are scoped to 15 minutes. This way if an attacker were able to get those creds in logs or they were copied and pasted into a public paste bin, they were no longer valid by the time someone came across them.
It's all about reducing scope.
The other thing I've noticed is that the team that sets up the CI/CD tooling usually is someone that set it up in their spare time to complete a goal and suddenly it is infrastructure that is weight bearing. You want to make sure that it is maintained like the infrastructure it actually is. That means dedicated system administrators, looping security in, understanding the flows, understanding who has access and how, and what ACL's exist, how execution happens, understanding what kind of access it has.
The lack of dedicated team/updates after it has been configured is killer. Old Jenkins versions are vulnerable to various issues that allow remote code execution in the context of Jenkins itself, so even if your pipelines only have time limited tokens, generally Jenkins itself has access to the keys to the kingdom (or runs on EC2 and now I've got access to the IAM instance profile keys).
Regularly updating means that pipelines will likely break, someone is relying on that old feature that is now gone in the new version. It requires dedicated engineering time to feed the CI/CD system. That is ignored far too often.
Here's ways that the team I was on broke in using CI/CD:
- Pipeline logs were sent to an S3 bucket, which was public and we found a URL an engineer pasted into an open source ticket (the team had helpfully base64 encoded all the secrets in a debugging build, which was also in the open bucket)
- CI/CD service was outdated and we could bypass the authentication it required giving us access to update pipelines
- We could create a new pull request on an open source project that was using an internal CI/CD tool, and steal all the secrets
- Found credentials for CI/CD in the source code to trigger a build on a downstream project (creds were not scoped, and had full access)
- Figured out CI/CD tooling was not tied to corporate AD, so guessed user accounts based upon devs commenting on open source and one of them re-used a password found in a previous breach
- Found creds for a Docker registry, uploaded our own build containers that exfiltrated all of the secrets based upon their own container so they didn't even notice
- CI/CD re-used the same host for executing privileged builds as un-privileged builds, and branches were automatically built. Found an abandoned branch, pushed to it, and broke out of the container due it being privileged, and then waited for the privileged build to start to steal the environment variables for that container
CI/CD literally is remote code execution as a service. Treat it as such.
> CI/CD literally is remote code execution as a service. Treat it as such.
That sums it up. The code in question is just doing the job it claims to do. CI/CD services literally have forms that allow you to run direct shell commands. If a highly paid engineer doesn't understand the code they are running, then it's time to fire them and get someone in place who does.
> the team had helpfully base64 encoded all the secrets in a debugging build, which was also in the open bucket
It's laughable that we still use base64 in any sort of security context (username:password), because the only people interested in decoding it can do it effortlessly. It almost feels like a weird form of procrastination, where we know what we're doing is wrong, but we're just too damn lazy to do anything about it.
Actually, in this case the CI/CD software would helpfully mask anything that it knew was secret, in an attempt to make the output it displayed "safe".
So to bypass that devs base64 encode the secrets (in this case the output from env) so that it is displayed and they can use it to debug that the right environment variables are set...
I regret the flippant tone, but, seriously, yeah: the other CI brouhaha this week was Homebrew breaking compatibility and no one noticed because they had their iOS CIs setup on autopilot, didn't see the deprecation warning in the terminal because, well, who the heck reads the entire stdout from a CI run?
So meant it more as treat it like you do pulling in a dependency: a black box that constantly requires validation and may break your whole stack of boxes at any point
p.s. its dumb, but, thanks for coming back here and conversating. at a real low this week for sundry reasons outside the news headlines, having a small, normal, discussion where someone assumed good faith even after I failed to make a full effort to participate positively, was a ray of sunshine.
Thanks for the reply! With that CrowdStrike founder Dmitri Alperovitch quote in the article
> Compromising and introducing a back door into a build environment such as TeamCity is the holy grail of a supply chain hack. It can allow an adversary to have thousands of SolarWinds-style back doors in all sorts of products in use by victims all over the world. This is a very big deal.
it read like this was much more widespread than I'm hearing now. The NY Times has even gone back and edited the title, replacing "Russian" with "widely used." I hope the editors have another take at this before it gets republished on all the other news sites ...
You're right that it's not the way to divulge important information, but (barring more reputable reports) I'm inclined to believe him. It's a somewhat regular occurrence for employees to comment on these things on HN, even if slightly misguided.
First, there's a WSJ article out now with some more details that appears to back him up[0]:
> Investigators believe that the SolarWinds hackers gained access to a TeamCity server used by SolarWinds to build its software products, but it is unclear how this system was accessed, according to people familiar with the matter.
Further, some rudimentary OSINT reserach indicates he is, in fact, a JetBrains employee. People can do their own sleuthing if they're inclined to verify this; I have no desire to dox the guy publicly.
The statement was already saying that one of the attack vectors was a bad configuration of TeamCity, the only difference was that it was in PR speech to protect JetBrains from false allegations.
> SolarWinds confirmed Wednesday that it used TeamCity software to assist with the development of its software and was investigating the software as part of its investigation. The company said it had yet to confirm a definitive link between JetBrains and the breach and compromise of its own software.
This is a very big "may". Reads like they are simply basing these allegations on the Russian founders part.
it is unbelievable the irresponsibility of the NYT to not provide more substantive evidence before going to print with this "may be", particularly with this title. The reputational hit alone will cost Jetbrains millions including across unrelated products (which they encourage by fluffing out this piece with Jetbrains' customer list without regard as to whether or not they use TeamCity).
Most people familiar with JetBrains (like software and system engineers) will likely read through the lines and simply ignore this article, given the good reputation of the company. The worrying part is that this type of article is destined for the less techy reader like the higher ups approving software purchases.
The lack of evidence feels like they got fingers pointed at them just for being... Russian?
Not that much surprising at all, Bloomberg with their "Chinese chip" story already proved that hit piece which will bring many eyeballs can be published by big "trustworthy" newspaper without basically any evidence and without significant consequences if evidence will never appear.
I still think Bloomberg should be sued for that. if due process finds that they had justifiable basis at the time, then fine. but if not this is grounds for libel. we can't have our top publications running around randomly throwing out accusations with zero consequences.
freedom of the press is important to me, but they also need to hold up their end of the bargain and have integrity and responsibility for their platform.
Agreed. Freedom of speech is very important to me, but I believe there's a big difference between an indivual speaking from their heart, and a corporation spewing nonsense.
Edit: To be clear, I don't think the solution is to silence an entity's publications. They do have the right to speak, but when they make unfounded accusations they shouldn't be surprised when they get sued to kindom come. We all have the right to speak, but the rest of us also have the right to defend ourselves and kick your fucking ass if you've misrepresented us with your speech in a public way, because that's libel.
I suspect TC may have been leveraged but it doesn’t mean it’s responsible for every horrible misconfiguration that can occur.
Case in point, I have used TC to gain AD administrative privileges before because the idiot who set it up ran a build agent as a domain admin so it could get access to a locked down signing cert. I just created a new build to add me to the right group and ran it on that agent.
These things are really trivial to find and exploit. Also the build agents will obtain and run almost any untrusted software and leave it on disk quite happily for when a later build comes along.
Here's what we need to be thinking instead of getting defensive or extra-optimistic:
If high-end security firms and fairly diligent government agencies were infiltrated, why would we think that smaller dev toolchain organisations not founded as security organisations will somehow be less likely to be targeted and become a vector for introducing supply chain attacks. Sophisticated attackers will go after any soft underbelly or pore they can find, and there's no reason not to believe they'd put significant effort into quietly abusing Jetbrains security just like they did with Solarwinds. I'm less worried about the "Russian" red scare mention other than it may give non-US organizations a few more opportunities to inject badness that we can't get visibility on.
Bottom-line is that it would be the holy grail and are we treating it as the high priority target it is? Having worked for dev tool companies in the past, I know they are a lot more worried about innovation than about their own internal processes.
Does it seem odd to anyone else that the article begins with a big Russian flag? That seems to be implying some sort of official Russian involvement. To me it seems much more likely that Solarwinds just left the username and password as admin:admin, and any detail about Jetbrains are kinda irrelevant
@mods - I changed the title from the original "Russian Software Company May Be Entry Point for U.S. Hack" to be more clear, as "Russian Software Company" felt vague and linkbait-y to me. Apologies if this violates the "No editorialized titles" policy.
Most of the software developers are still in Saint-Petersburg, Russia. Russian founders opened a Czech company because being Russian, they could not easily reach world markets.
The title and lede state they are "Russian-Owned" and a Czech company. Perhaps NYT edited title after publishing or original poster misread the title.
Russian-Owned Software Company May Be Entry Point for Huge U.S. Hacking
Russian hackers may have piggybacked on a tool developed by JetBrains, which is based in the Czech Republic, to gain access to federal government and private sector systems in the United States.
"Russian-owned" is still largely misleading. The company is owned by Russian speakers that lived and studied in Russia before starting JetBrains. Most (if not all) of them have two citizenships by now, one is Russian. The company is not owned by the Russian state and has never been.
In Europe, words like Russian or Czech typically mean ethnicity. Even if you live there for 20 years, learn the language and get citizenship, people likely won't call you Czech. Maybe you can be called Czechian (if it's a real word) or Praguer since those don't imply ethnicity. I don't live in Czechia though, so take this with a grain of salt.
> The company is not owned by the Russian state and has never been.
This is true however, and NYT did (predictably) poor job here. Unless there is any indication that JetBrains were involved in the hack, mentioning "Russian-owned" in the title is misleading.
>In Europe, words like Russian or Czech typically mean ethnicity.
Well, then, suddenly all those fines on some obscure Russian (alright, alright, half-Russian) company called Google all make sense now! Just another part of the sanctions!
My issue with the “obscure” comment is what it suggests to the non-dev reader. It is easy to think that “Obscure” companies can be pushed around more easily by state actors. That their product has never been critically evaluated for security issues.
There may well have been an issue emanating from jetbrains. I don’t know.
But when you combine these insinuations with the fact that the company is run by Russian nationals (quite different from the Russian government!) you simply get stronger insinuations, not facts.
To anyone outside of tech it's completely obscure. No one besides developers and people who work with them know what the heck JetBrains is. The same way almost no one outside of tech would know what "Vim" is, even though to us it seems like a household name.
That's still terrible reporting. Would it be fair to call Salesforce or PeopleSoft obscure? I've worked in tech for over a decade and I barely know what they are--only that they're giant. I'm confident most people have never heard of them.
It's amazing to me how many programmers get shook to the core when something that seems so obviously important to them is actually not a big deal to the rest of the world.
I think the characterization is what's surprising. I would imagine there's a major supplier for petri dishes out there that everyone in the medical field knows of, but I've never heard of them and wouldn't call them "obscure" just because it's niche. Using that criteria every software company outside of Microsoft or Apple would be considered obscure.
> It's amazing to me how many programmers get shook to the core when something that seems so obviously important to them is actually not a big deal to the rest of the world.
> The characterization really, truly, does not matter.
I don't think that's a fair sentiment. Within the niche that Jetbrains occupies it is one of the most well known players out there, and so it truly is not obscure. It was an unfair characterization. Furthermore, the reporting provides no evidence to support its very damaging claim. To those of us who are familiar with configuring these types of systems, we know very well how easy it is to leave a single permission unattended to. The damage done by this type of irresponsible reporting is serious. Personally, I hope they get their pants sued off for their complete lack of discretion and disregard for journalistic integrity.
> Within the niche that Jetbrains occupies it is one of the most well known players out there, and so it truly is not obscure.
Doesn't matter. I'll repeat that it's still amazing how many programmers get shook to the core when something that seems so obviously important to them is actually not a big deal to the rest of the world.
> Furthermore [...]
Unrelated to my post :). But I hope investigations show that TeamCity was indeed misconfigured and that nothing nefarious is at play w.r.t. their tools.
Yeah, you're right. My diatribe near the end was something I had to get off my chest, but looking back there was nothing in your comment that it was responding to. I should be more mindful where I tote my baggage around, lol.
I had a flat mate at university who was a music major; I once mentioned that I'd discovered an obscure Italian composer that I liked-- he very quickly made it obvious that the composer was not obscure and that he was one of the most prominent composers of a certain style in a random decade. Obscure is somewhat relative, but it is usually taken in the context of people who know _something_ about the subject matter. My position is that the composer was indeed not obscure, I was just ignorant of the wider culture and knowledge of musical history.
She’s setting up a scene for a play. Just from those couple line I’m writting whole thing off as same neocon Russian witch hunt we saw in recent years.
I'm with you on that, those words serve to diminish the work and reputation of JetBrains even before the ~average reader understand who they are and what they do.
Like "some idiot, edoceo" seeds that I'm a fool vs "my CTO, edoceo" (of course the truth is in the middle)
distinction without a difference as far as the new york times is concerned. i'm saying we as developers shouldnt make that our number 1 complaint (which it is, given responses to her twitter) instead of focusing on the irresponsibility of the NYT publishing a "may be" article with 1 line of speculation and 15 paragraphs of context/fluff and 0 lines of evidence or findings.
The teamcity application is a java application that can be easily analyzed for security threats. Nothing stops you from decompiling it and check the code. This story looks very much like a scapegoat for SolarWinds' poor security practices.
yeah.. and next breaking news - Microsoft and Google also may be entry points for U.S. Hack because Solarwinds used both Windows laptops and Android mobile phones.
I wish tech reporting is written by technical folks, or at least proof read by tech experts.
> I wish tech reporting is written by technical folks, or at least proof read by tech experts.
This comes up for journalism of every single industry.
“Briefly stated, the Gell-Mann Amnesia effect is as follows. You open the newspaper to an article on some subject you know well. In Murray's case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward—reversing cause and effect. I call these the "wet streets cause rain" stories. Paper's full of them.
In any case, you read with exasperation or amusement the multiple errors in a story, and then turn the page to national or international affairs, and read as if the rest of the newspaper was somehow more accurate about Palestine than the baloney you just read. You turn the page, and forget what you know.”
Is there any published technical evidence to look at which actually implicates JetBrains? CVE or a POC? Publication of the poisoned build of the CI software? Indicators of compromise?
They probably already had access to the network and just used TeamCity since that was SolarWinds build process. Or someone really misconfigured things to allow external access.
What evidence do they have, or is it process of elimination, like source code built normal binaries locally, but malicious binaries when built in CI? Hope they have some VM images of the CI boxes for forensic analysis.
