Not a company I'd put much actual production onto. Imagine if AWS had a hacker running around with total root access, able to reset MFA tokens to their own etc with no notice to customers. I'm not even sure such root access exists on AWS.
Look, we are random internet people, and it's a "me vs. you" scenario, but as someone who worked at Linode in 2012, we were a small company, that all worked out of one office, with like 3 admins at the time. Yes, there were various hacks. Yes, there were silly vulns, but positing that one of the employees at that time stole bitcoin is something that I won't stand for.
Could it have happened, sure. Do I think that it was an inside job? No, not at all. 99% of the people there at that time thought Bitcoin was insanely dumb, and I suspect most of us still do.
As I noted elsewhere, there's an opportunity to do the right thing when someone comes to you and says look, someone is coming in on the control plane and resetting my server passwords.
And yes, that includes looking at your staff especially when bitcoin is in the mix as its less traceable to a person.
For some reason, for year after year, there was this pattern. No problem, we have good security, oh wait, we've been rooted for months. Or someone is coming in on the staff admin plane and taking all sorts of action.
It could be outside hackers sure. But linode never seemed that interested in sorting things out.
The takeaway I had was that you might not notice if a staff person OR hacker was messing around.
When you're in Linode's position, running people's private and infrastructure, you have a very finite amount of grace. Individual employees at linode had an incentive to violate the company's security. When security violations happened that could plausibly have come from employees following their incentives the company consistently failed to assure customers it was making sure the security issues were being addressed. In my opinion this at least shows a wild disregard the well-being of customers and I think it's pretty normal to treat that kind of wild-disregard as malicious (even though it may not be).
This followed the pagerduty hack. We don't know who else was getting hacked either - these were to high profile ones.
So this just raised the question - what's up that they don't take serious issues seriously? With bitcoin there have been a ton of insider issues with how "trusted" infrastructure providers and exchanges handling bitcoin so that was the question.
Even if external hackers, they just got hacked over and over.
If the known facts are consistent with your staff stealing Bitcoin and the reason it can't be confirmed whether or not this happened is because your staff fell short of industry-standard security practices, I think it's entirely fair to say that that might be what happened. Put it this way: from the perspective of someone on the outside, if your staff were stealing Bitcoin this is exactly what it would look like.
As someone who commented on Linode hacks earlier, I can vouch for lbotos having worked there and feel the same way as they do. I don't believe any of the attacks were an inside job, because I don't believe anyone would have done that and if they did they knew how not to leave a trail behind.
Backing up both Tim and Lee here as a former who overlapped with both of them. I had many issues with Linode as an employee. The idea that anybody I worked with at the time, many of whom I don’t get along with because I was even more of an asshole then than I am now, the idea that any of them would pinch Bitcoin off a Linode is so off-base it’s laughable. It simply didn’t happen. Period. If you believe it did, your logic in getting there is no different than that of political conspiracies that are common today.
I remember that rash of Bitcoin thefts and it was all careless behavior by the Linode owner becoming a secondary consequence of a primary employee compromise, I think. As in what happened to Twitter. Think “admin panel compromised, external actor searches for Linodes known to participate in Bitcoin, methodically compromises them one by one, finds poorly stored wallets and drains them”. That intruder very obviously knew what they were after, if memory serves, but this was almost ten years ago.
Seriously. Linode did one thing well and it was hire (mostly) good people. The comms around security incidents could always use improvement, and I think that led to the loss of trust you’re seeing here. I don’t think it’s just Linode, either, I think a lot of the industry is overly discreet when it comes to what to say publicly about events like this. We see the same with journalism: a lot of methods in reporting are trade skills and most people don’t understand the news gathering process, which leaves room to fill in the gaps with conspiracy. So it is with security, too.
I’d back your speculation, Tim: there were maybe two people, definitely one, maybe two, who could both perform the crime and hide it. One’s an unsavory person to interact with if he doesn’t like you but ultimately ethical and a force for good at his core. The other runs the company. Convince me that either of them did that and you may as well convince me the Earth is flat.
I think in 2012 the tech community's sentiment was actually that Bitcoin was really cool and definitely useful, since it was new and did things in a practical application that we hadn't seen before. It's interesting that hindsight clouds that, it has been viewed negatively for years now but it's not that old yet. (Either way, your main point stands of course.)
> I think in 2012 the tech community's sentiment was actually that Bitcoin was really cool and definitely useful
I don't think there was ever really a consensus on this. Lots of people (myself included, but also quite a few friends) always thought Bitcoin was just kinda useless. It's just that in 2012 there were comparatively low stakes (i.e. no massive energy use, not yet massive amounts of people pouring money in it, no massive amounts of "crypto snakeoil") that it just wasn't worth commenting on.
I find this exceedingly hard to believe. Around 2009-2010, btc was definitelynot an "asset" (like the bafoons try to treat it now).
It *was a currency*.
Many still believe in the idea that (certain, less well know) crypto can be used as a real currency, but unfortunately the public severely tainted it with ideas of 'being an asset'.
This comment reads as someone who is more aligned with the public's (HN) perception of modern crypto, rather than the use of it pre-2010.
I think the idea of “hodl” bitcoin wasn’t there. One thing that changed in my perspective is that bitcoin transactions were always destined to be more expensive than I had dreamed. In my mind, I thought transactions would be fast and free of cost. In reality, there are reportedly fewer than ten thousand full nodes.
Everyone thinks of bitcoin in terms of “how many USD is it?” I don’t know what the solution is but as long as we think of bitcoin as a perverted asset like housing - apparently people once again believe we will not allow housing prices to fall to any significant degree - there is no reason to use bitcoin as a currency. With so much speculation, the price is too volatile.
I don’t know what the solution is but I believe transaction costs should be minimal if not zero. I don’t know how we will achieve this but apparently there are other projects that try to get much closer to zero transaction fees. I think that is the future
There were plenty of us who knew – and said – that "cryptocurrency" was borne of technical, political, economic, societal ignorance when it started. Now it's just more obviously terrible.
The sentiment I knew at the time was a mix of excitement at the cool new tech, skepticism of the usefulness of it (my camp), and drooling over using the GPU you already had to make easy money.
nope, I remember starting my first tech job in 2013, and the only people in tech who cared about it were libertarians, which was a very small subgroup of tech
"I think money is insanely dumb, and I suspect most of us do ... So it's ok if someone steals money."
I think I get your sentiment(?), but I'm uncertain of whether or not it matters how individuals value 'success' or 'currency' when it comes to personal property.
To extend to the logical consequences, I am not sure if people on HN would agree that personal property should be non-existent.
No, sorry that I was unclear. My point was that it's my belief that those of us employed at that time did not see value in bitcoin, so we had no motivation to steal bitcoin that would be a small fraction of what we were getting paid.
Now if $some_duder_was_really_into_bitcoin was also on the staff at the time, then sure, maybe they would risk their job to steal some bitcoin because they thought it was cool to do hax0r things with cyberpunk money. I'm not aware of that person existing.
A 2012 Bitcoin hack victim was none other than a lead developer of Bitcoin. Back then, they ran a Bitcoin faucet on it that gave out a paltry 0.25 Bitcoin at a time.
I never bothered to jump through those hoops for like a dollar (now about US$10k):
Ensuring the security of our platform is our top priority. We maintain a strong security policy and aim to communicate openly should it ever be compromised. Thus, we are posting to describe a recent incident affecting the Linode Manager.
Here are the facts:
This morning, an intruder accessed a web-based Linode customer service portal. Suspicious events prompted an immediate investigation and the compromised credentials used by this intruder were then restricted. All activity via the web portal is logged, and an exhaustive audit has provided the following:
All activity by the intruder was limited to a total of eight customers, all of which had references to "bitcoin". The intruder proceeded to compromise those Linode Manager accounts, with the apparent goal of finding and transferring any bitcoins. Those customers affected have been notified. If you have not received a notification then your account is unaffected. Again, only eight accounts were affected.
The portal does not have access to credit card information or Linode Manager user passwords. Only those eight accounts were viewed or manipulated -- no other accounts were viewed or accessed.
Security is our number one priority and has been for over eight years. We depend on and value the trust our customers have placed in us. Now, more than ever, we remain committed to ensuring the safety and security of our customers' accounts, and will be reviewing our policies and procedures to prevent this from ever recurring.
---
I won't argue Linode is blameless here, but seems like the only reason it had such an outsized impact was because the 8 customers who were targeted evidently didn't do much to protect their assets from someone gaining unauthorized access to their servers--which is always a possibility with any publicly exposed server with or without a breach of the service provider being involved.
Hi. This is my comment you keep linking to. Your understanding of what happened is flawed. I do not have signs that Linode was rooted in that compromise. The signals I do have is that they had their database compromised, and likely secret key material. That allowed attackers to crack the hashes offline, and then authenticate using MFA.
IMO, it's plain wrong to categorize that one as "getting full root control plane", where it was instead the compromising of individual accounts that may have had no access to the resources on an account.
That’s a little over the top, eh? They disclosed within 4 days of discovery and implemented better security controls all over. They are probably the only major retailer with chip and pin payments in the US, for example.
If you think they were unique or egregious in terms of 3rd party access to networks, i am afraid that you will find reality disappointing.
Should that same existential penalty be applied to every company who had Log4J running in prod a few months back? That was a much more widespread root compromise...
Not comparable. Log4j was a vulnerability in the software, not leaving their shit open and and putting all their cash registers on an intranet available to the internet.
The one thing I don't understand is if that many coins were stolen and every transaction is traceable shouldn't there be a trail? The owner has 124 million reasons to find those coins. Is the ability to track past transactions not as possible as it seems?
You can track, but large amounts merge with small and disperse as they hit brokers etc. If your claim to someone is that their BTC is 0.01% stolen, it's not so strong. Faster you act, more you can do
somebody please explain to me how is it possible that the owner reports losing $124 million, then they casually mention in the reply that: no problem, I'll just cover it with my own money ...
(another recent story on ether hack had the same "resolution" the organization just chose to replaced the losses) ...
where is that money coming from? does not seem real
That’s what it would be worth now. Was worth like 1/10000th of that back then. They probably covered it with their previously earned holdings that could have been another fraction of that.
I hate to say it but I recently moved away from Linode after their /64 block in Frankfurt was banned by all Google services. And even though all their kubernetes nodes have a public ipv4 address they were somehow unable to fallback on this when their ipv6 didn't work.
And when I suggested this to their support they acted like I was crazy and said there is no way to switch between ipv4 and ipv6. Well I don't work in networking but I do work for a major telco and I know our networking guys could have done that routing change, easy.
We blocked the entire of Linode AS63949 ranges because we were getting attacked from random owned nodes and it was tripping our IDS constantly. Just got fed up with it in the end and decided to hose them.
To note, we have had problems with AWS blocking random addresses as well where we've had staff abroad.
I never said it was "nefarious", I'm saying it was badly handled and it brought down all services in an entire location for days, with no fallback to ipv4, no status communication from Linode.
It's not malice, it's just incompetence.
Why the block was banned I have no idea and I don't really care, it can happen and they need to be able to handle it.
I literally had a support case asking what happened, no response, eventually I had to say "look, I'm moving now because all services are down, please tell me what is going on", no answer, I had to move to get my services back up.
The January 2016 thing was them finally acknowledging the attack that had happened many months prior, after WPEngine gave Linode the opportunity to announce they were hacked after they were also compromised using the same vector that hit PagerDuty. If Linode had declined, WPEngine was going to do it on their behalf. I couldn't convince the powers that be to make the same demand months prior, even though I was confident, so if WPEngine hadn't pushed the issue I don't know that Linode would have ever disclosed.
But that doesn't matter anymore. This was nearly a generation ago in tech companies, and they are now part of a bigger one.
You can’t just imply that the linode staff assisted or were involved in stealing crypto currency from their customers without actually providing any evidence.
My metric though is this: Someone has admin level access to full root on control plane and does things to customer accounts they don't want. A customer complains that the control plane has been used to reset things / asks for logs / etc.
That's your opportunity to identify a root level control plane hack, disclose it, do the password reset things, remediate and move forward.
OR - You don't disclose things till much later, you deny things, you provide scrubbed logs. Add in the bitcoin angle where this type of story is pretty common (trusted entity runs away with coins because they are easy to steal) - and couple that with how they handled the reports -> as I said in my comment, I'd be careful putting sensitive info onto that platform.
I ended up dropping Linode as a result of the hacks, after having been a customer for years, both because of the nature of the hacks, and how they communicated.
This was the same era that Linode had the compromises listed. It's a pretty apples-to-apples comparison, though I agree it has been almost a decade since.
A lot of bitcoin theft in 2012 (maybe by their own staff?)
2013 some kind of cold fusion / HTP hack
Another CF / HTP hack here.
2014 brought the MySQL server no password stuff.
2015 ish some kind of total root compromise?
You can get a feel for all this here including the denials / lack of notification.
https://news.ycombinator.com/item?id=10845985
Maybe 2016 same issue?
https://www.zdnet.com/article/cloud-firm-linode-resets-user-...
Not a company I'd put much actual production onto. Imagine if AWS had a hacker running around with total root access, able to reset MFA tokens to their own etc with no notice to customers. I'm not even sure such root access exists on AWS.