I'm fairly confident that Linode has been compromised since July, if not earlier. PagerDuty moved off of Linode after an incident in July. We've been under strict gag orders from legal about that incident until today when Linode finally announced their compromise. Really, the only way I can see that this attacker could have gotten in the way they did (they logged into our Linode Manager account on the first try using a username that wasn't used anywhere but in Linode Manager, using valid 2fa and valid password) was if they had access to the Linode Manager database. I'm pretty sure that the recent WP Engine compromise was achieved through the same attack vector, too
This was my post from SomethingAwful, it was copied/pasted here. No idea who copied/pasted it.
Rather than writing up a huge reply with more info, I'll link to another reply with more details on this same thread since someone else already did the writing: https://news.ycombinator.com/item?id=10845985
Interestingly, while I know for a fact that you're correct. When someone asked about this on #linode the ops immediate reaction was to deny it (as it was every other time they got hacked).
As a former Linode employee, I find all of your comments very interesting. But if it's true that you're being prosecuted, why are you incriminating yourself here?
All of the people involved have been very open about their identities and what they've been up to. It's sort of their MO. Pure speculation on my part, but I imagine it's basically a giant middle finger to everyone else.
Difficult question to answer. There was a complete lack of "technical" evidence against me (e.g .bash_history files, wiretaps).
The only evidence the prosecution had against me were a list of compromised sites and several coldfusion 0days I had in my possession.
They could never prove that I generated the list of compromised sites, but the judges felt that the possession of said list was enough evidence to convict me.
We would've appealed but there was no point as the sentence was essentially nothing.
Since I'm not really planning to look for a job, my main worry was potential visa issues. But I haven't had any troubles visiting the few countries I do need a visa for.
The other side to that is that if you appeal, and lose, you can come off worse. So even though a criminal record is "something", depending where the lawyers feel your chances are, it may be best going along with it.
I find this very interesting because I think it brings up a specific question. If this is actually from July, and they knew about it, why would Linode choose this specific time to announce the breach publicly? It seems very curious being that they are already getting punished from the DDoS attack and if they waited 6 months they could have waited until the dust settles a little bit.
Unless, of course, the group involved in the breach are also the ones unleashing the DDoS attacks. Which would also make me think there has been communication between the group and Linode, in contrary to what Linode stated.
Could be, but the timing still seems really weird to me. If they did an investigation like they said they did, I just don't see why you would come out with it right now. Because from Pager Duty post it seems that no one could do anything to make them disclose it sooner.
I can't speak for the other folks that were compromised this way, but we decided to just cut our losses and move on at PagerDuty and spent the 30 days after the compromise migrating everything that was running there over to Azure. No point in putting pressure on a company that stonewalls you.
That's a good point. Not worth your time for a company like Linode that doesn't really care about its customers. I think people mistake the quick support responses to basic questions as them caring, but when it really comes down to the important things like security and communication during a crisis, it's clear that there is a huge lapse from the leadership level down. Someone in this post wrote about how they stopped $10k worth of Linode service a month and no one tried to get them back or retain them. That was very odd to read, especially for a business of Linode's size ($22 million in revenue isn't that big where you can shrug off $120k/year.) Par for the course it seems, and everything is starting to make sense. Thanks for sharing your story and sorry that happened to your team.
I know we left them 4 years ago because how they implement bandwidth caps on private IPs, which a few years later another company also had the same problem with and nicely wrote it up on their blog: https://docraptor.com/blog/gone-in-60-seconds-how-we-moved-f...
Linode's attempt to keep us (spending $5k a month at the time, but we've grown substantially with AWS now. Linode were at $22m in 2011, so we'd have been 0.3% of their total revenue): "We will certainly be sorry to see you go."
That may just be because they were fed up of us after we opened 27 support tickets about the same networking issue over the course of 11 months though.
When I worked there support was under intense pressure to respond to tickets quickly.
Not nearly as much pressure to respond correctly.
To clarify and reiterate--employees who responded to tickets quickly were praised, even though there response contained half truths or outright falsehoods. If someone took 15-60 minutes ( or more ) to deep dive into an issue for a real fix for a customer, they were shamed and got a talking to.
> If the Linode Manager database only stores the hash, how would they know the password?
Hashes can be turned back into plain text, it is just computationally expensive to do so. Hashing only slows down an attack (and or increases the cost), it doesn't not mitigate one. In particular if the hashes aren't salted then a rainbow table is an extremely effective way of breaking all of the hashes concurrently.
The main method of doing so: Generate the hash for every combination of typable characters up to a given length (e.g. MD5() A-Za-z0-9 & specials up to 8x characters).
This can be mitigated using a more computationally expensive hashing routine (or increasing the work factor on a less computationally expensive one) and salts.
But given enough time OR computing power, all hashes will be broken. AWS makes breaking hashes a lot cheaper as you can bid on spare capacity and perform the operations relatively cheaply.
This only works if the input password has low entropy. You would think that people using Linode are savvy enough to be using long, randomly generated passwords.
12 character random strings are an absolute minimum for a secure password, because brute forcing and tabling start to become impractical. Longer strings are even better.
I wouldn't consider an 8 char password secure, no matter what the entropy is.
A good secure password hash should be generated with a salt to slow down the process, in addition to using strong (slow) KDF function like scrypt. State actor like NSA could have store all possible 8 character combination today in a massive storage facility. God knows. But the size of such rainbow table is only effective if there is no salt, as with salt the hash is now different despite the underlying password is the same, thus there will never be enough storage if salt is present.
But the most effective "rainbow table"-like table is a look-up table with the followings:
* leaked password in plaintext, associate with email and any ID (forum username??)
* hash all of those passwords without salt
* hashes (with salt) of known leaked passwords (you try pas$w0rd and found a match for some hash with salt) - this only works if your attack succeed.
If you do a quick count you won't be surprise most passwords are fairly short and simple. If two complex passwords appear to be very similar, you can assume with a good probability they are used by the same person. You can learn some private data from just looking at password (e.g. birthday, pet's name, door number, company they worked for, sport team they root for, which many turn out to be the crucial hint or actual answer to security questions.)
I have never opened or downloaded any leaked data and don't know if it legal for use at all, but the black market probably has over petabyte volume of such data available.
It would be very interesting to see the whole world attack couple hashes per day. Imagine you go to a website, it gives you some plaintext, and you run a couple quick scrypt with random salt, and return the response. Now with a billion online users, run this every day once, you may end up finding one successful match of "this password == this hash with this salt" once in a while. But hey, that's what botnet can do...and then bitcoin!
If we presume the attackers had access to the system handling authorization, then the attackers introducing code to ship passwords offsite as users log in isn't really a stretch of the imagination.
That's true and great point. Let's play devil. You can't really be sure if Google engineer is sloppy and logging username and password on entry and then the SRE reading the log sees everything. I am not sure if their build system has plugin to detect such big red flag. Stacktrace is another place with potential leak of credentials. All of these are reasons when someone claims software is open source and auditable but they run the service themselves, it's really important to note you can't audit the actual server. They can log your username and password behind the scene while the client side appears to be 100% the same as the one on the server side.
What you describe is not rare, can be done with cross-site scripting. How it happens depends on the injection method (perhaps SQL injection).
If they've been more compromised than something like a simple SQL injection vulnerability, it could be something like added code to log passwords or post them off somewhere else.
This is at least the fifth time I can count that linode has been hacked, really? Maybe it's time to ditch that coldfusion stack?
Edit:
1.The bitcoin hacks, March 2012
2.HTP hack, April 15, 2013 (CF exploit)
3.Second HTP hack April 16, 2013 (Another CF exploit)
4.MySQL server that allowed anonymous logins (?!?!) January 19, 2014
5.This hack
I'm not counting their domain name and various other parts of their infrastructure provided by third parties being compromised, but if I was the list would be significantly longer.
Today's attack is some kind of unspecified or unknown breach involving Linode manager.
I guess the obvious commonality here is that all the attacks target the "soft" Linode layers AROUND managing deploys of Linux and Xen/KVM/UML rather than the "hard" targets of those widely used systems. This also happens to be the layer where Linode should be adding value (as opposed to the cheaper VPS providers out there) and I think it's increasingly troublesome that they continue to have such severe security issues.
Is this company (CEO - Christopher Aker) not investing in security staff, security training, best practices etc, or are they investing tons and just getting breached because they host so many sites? Unclear. But it's easy to imagine it's the former, from the outside, given all these incidents.
A customer warned Linode team about the exposed CF folder. CEO aggressively shrugged it off. "That doesn't matter, it's nothing, that's a non issue." Dev who was a bit of a suck up parroted the same telling support to shut up about it. This was six months before HTP happened.
I'd avoid VPS providers in general, but AWS is on a whole different level than linode. They actually understand what they're doing well enough to do live xen patching etc.
But yeah, people get hacked through their hosts all the time. Best approach is colo with minimum access for the dc staff.
That's been my recommend for a long time. Plus, I liked obfuscating with unusual CPU choices and network guards (esp for protocol layers). Worked wonders with about no effort outside setting up guards. Opponents throw so much x86 shellcode at your Alpha, etc boxes while never quite getting stuff to run.
I know of at least one other of the alleged HTP members who is not only not arrested, but still actively involved in the information security community.
Then there are other personalities that went dark, whom are presumably also not arrested.
Unless Thomas Asaro can name names, that was a bluff.
Been working on (re)writing things in Python since I started. It takes a while, but I think everyone recognizes that the CF codebase is difficult to maintain. The good news is that significant progress is being made and we're still doing routine audits of the existing stuff.
>we're still doing routine audits of the existing stuff.
This doesn't work, auditing coldfusion code is impossible without auditing the entire platform. The whole platform is so full of bugs and strange behaviour that it's actually impossible to produce secure coldfusion code.
"Effective immediately, Linode Manager passwords have been expired. You will be prompted to set a new password on your next login. We regret this inconvenience, however this is a necessary precaution.
"A security investigation into the unauthorized login of three accounts has led us to the discovery of two Linode.com user credentials on an external machine. This implies user credentials could have been read from our database, either offline or on, at some point. The user table contains usernames, email addresses, securely hashed passwords and encrypted two-factor seeds. The resetting of your password will invalidate the old credentials.
"This may have contributed to the unauthorized access of the three Linode customer accounts mentioned above, which were logged into via manager.linode.com. The affected customers were notified immediately. We have found no other evidence of access to Linode infrastructure, including host machines and virtual machine data.
"The entire Linode team has been working around the clock to address both this issue and the ongoing DDoS attacks. We’ve retained a well-known third-party security firm to aid in our investigation. Multiple Federal law enforcement authorities are also investigating and have cases open for both issues. When the thorough investigation is complete, we will share an update on the findings.
"You may be wondering if the same person or group is behind these malicious acts. We are wondering the same thing. At this point we have no information about who is behind either issue. We have not been contacted by anyone taking accountability or making demands. The acts may be related and they may not be.
"The security of your data, the functionality of your servers, and your confidence in Linode are extremely important to all of us. While we feel victimized ourselves, we understand it is our responsibility, and our privilege as your host, to provide the best possible security and service. You can help further enhance the security of your account by always using strong passwords, enabling two-factor authentication, and never using the same password at multiple services.
"We sincerely apologize for the recent disruptions in your Linode service. Thank you for your patience, understanding and ongoing trust in Linode."
Sadly I trust them as far as I can throw them, and we moved everything important from them to AWS a few years back. We had left a few static sites there but after the shenanigans over the holidays, we're moving our remaining stuff.
Sad. We spent >$10k/month with them for a while, before their shit started falling apart. They didn't appear to care at all when we left, so I suppose they have an awful lot of large customers who just don't mind their stack disintegrating beneath them every sixty days at most.
Well, at linode you can't have a structure that is immune to failover, as they have single points of failure within their infrastructure, apart from anything else - all their London kit for instance lives in Telehouse East, in a few adjacent racks.
Once we'd done the initial up sticks and move to AWS, our first priority was to use their redundancy and failover to the fullest (six months of sleepless nights due to linode made this rather front and foremost in our minds) - so nothing that's happened at AWS has ever been anything more than an inconvenience - we've managed five nines since the move - before, we managed one.
Only one 9? Even through this crap during the holidays I've managed 3 9's on my service hosted on several servers in Linode Dallas (the most hard-hit region in this DDoS attack). I would have moved to AWS by now if Linode didn't have such cheaper bandwidth.
Yeah... our issues mostly arose from the fact that at the time, they were advertising 1Gbps node interconnect - which actually turned out to be 1Gbps HOST interconnect, with the nodes actually throttled down to 50Mbps. We use memcached extensively, and this was absolutely crippling for performance. It didn't help that they furiously denied that they were throttling until we demonstrated it beyond all doubt.
They did obligingly increase these caps when we begged them to do so, but at that point the writing was on the wall, and we kept on bumping into other weird and wonderful limitations and issues, such as the fact that someone running an intense job on the same host could bring our VPS's to an absolute crawl.
It really is a shame, as we desperately wanted to make it work, as we liked Chris's hands on approach (very much like ours), but ultimately our confidence in them was so eroded by the point that things started going genuinely wrong on their end that we had no choice but to leave.
As I said, we kept random small single-server stuff (wordpress sites mostly) there, as if you're not dealing with their networking, performance is generally OK - but the network limitations were an absolute clincher for us, and it was at one point literally every day that we'd find that one of their switches had broken, or we couldn't ARP IP's for no apparent reason, etc. etc.
ATL was much worse than Dallas in this go around. You did not get 3 9s in the last month if your service was located solely in ATL. They were completely null routed for the majority of 2 days.
Strange how this works. I moved everything to cloud providers because of availability issues. Now a few years later, I'm moving everything back to dedicated hardware in several different datacenters because of availability issues. Thanks, Docker!! <3
We are leasing hardware, and we are leasing a lot of OVH (in addition to a number of smaller providers). OVH does a good job, and if you build your infrastructure to load-balance around failure (which does not happen often), you'll hardly notice it. Often we get an email from OVH telling us that support techs have been automatically dispatched to a node which we hadn't even alerted about yet. Docker is basically a miracle drug for this kind of thing. Our hardware costs are less than half of what they were with EC2. Huge savings.
At this point I'm starting to wonder whether this isn't a competitor putting their investors money to work. It's otherwise utterly bizzare that someone would be so obsessive in damaging Linode. I really hope they make the details of the investigation public...
Linode, Github, Stackoverflow, Imgur, they've all been targeted. But what do they have in common? In a word: popularity. The core reason these sites are targeted is because it is impressive to others.
The source of this is typically two fold:
- For the lolz. Someone with a botnet just wants to show off, taking down something known gives them more notoriety.
- For a sales pitch. Someone has botnet capacity that they want to sell, and "I took down Linode" is a great way of demoing that to potential buyers.
The first one is more common when someone finds a new method of traffic multiplication and just wants to show it off (i.e. they trick a third party into DoS/DDoSing a target). The second one is legitimate criminal enterprises.
Github was also targeted by China to try to get them to delete certain repos that were unfriendly to China. Some people also saw it as showing off their power.
Sure wish they had sent out an email notification to users instead of a slashdotted blog post. Now the question is how long can Linode stand in the face of these sorts of hacks and network attacks in the face of stiff VPS competition.
(Linode Employee) Already got it covered, we are sending out an email to everyone in batches, but pushed out the blog first since it can be seen by everyone right away.
Be grateful they posted something to Slashdot at all. Back in 2013 they didn't do that for DAYS after the event happened and someone else posted what happened.
Not to throw fuel on the fire but grumbling ex-employees aren't surprised. I've heard a few stories about not investing in infrastructure or even bug fixes. "They wont even look at bug fixes with major support costs." I know it's hard to triage what to do when running a company but when employees are proud of their product a breach like this isn't surprising.
I'm glad to see that this information has now been publicly disclosed. In July 2015, we suffered a compromise at PagerDuty via the Linode Manager. I hope that we can provide a bit more of an official in-depth post-mortem of our compromise, but I'd be happy to disclose some of the details here.
Using the access gained within the Linode Manager, the attacker reset the root password on a few systems, and used Lish to gain root access. We were alerted to this activity and fully revoked the attacker's access within 60 minutes of the first node being compromised. Working with Linode support, we discovered which user account was being used and completely deactivated the user. We also isolated the VMs, and performed forensics on read-only copies of their disk images.
In our situation the attacker knew one of our user's passwords and MFA secret. This allowed them to provide valid authentication credentials for an account in the Linode Manager. It's worth noting that all of our active user accounts had two-factor authentication enabled. An interesting data point was that the user who had their account compromised was no longer in possession of the MFA secret themselves. Their cell phone had been reset (thus deleting all data) 8 months prior. The user could not log in to the Linode Manager if they wanted, so it was our determination that the key could not have been obtained from the user and was more likely on Linode's side.
We also have evidence from access logs provided by Linode that the attackers tried to authenticate as an ex-employee, whose username ONLY existed in the Linode database. It was absolutely unique and was not used elsewhere by the employee making the username an accidental honeypot. This was another piece of data supporting that Linode was the source of our compromise.
We immediately reached out to them not only to inform them of their compromise, but to assist them in investigating it. We were confident that the Linode database had been breached, and that the secret key used to encrypt information in the database had been compromised as well.
In addition to reaching out to Linode, we also worked with a third-party security firm to audit our work done during the incident. Likewise, around the same time we reached out to law enforcement to assist in investigating the attack. I believe our public disclosure includes this information[1]. This was in the middle of July 2015.
We did not get confirmation in July that there was a breach of the Linode Manager or any associated credentials.
In the end, we migrated away from Linode because of this breach (even before it was publicly disclosed) in Aug 2015. We also never were able to confidently disclose that Linode was the vector due to lack of confirmation from their end. While all of us who responded to the incident were confident they were the source, we now thankfully have the data to confirm it.
That's incredibly poor handling by linode, it also quite possibly resulted in many other systems being compromised via the same route or a similar one in the meantime.
Saw your tweet (https://twitter.com/theckman/status/684484772316360705) that linked to this post. Did a quick search to get your technical background and your LinkedIn profile states you used to work for Linode? I think it's important to share that info when you're telling your side of the incident. Your past relationship, if you left on bad terms, could play a role in your motivation to post.
I can absolutely understand your concern. I originally had it in my post, but removed it because I was worried it would detract from the details of our compromise at PagerDuty.
I worked at Linode for just under three years, and worked on quite a few different things there. I started on support and moved on to a development role (including writing ColdFusion). I left Linode on good terms. California is much more enticing than NJ, so I wanted to relocate. Plus I was interested in doing more of an Ops role, instead of working including customer-facing web applications. I'm still enjoying it. :)
I think there are lessons that can be learned whenever a company has some sort of security incident. This is especially true if they are willing to publicly disclose details of the incident. We've wanted to provide what limited information we had, but wanted to wait until we had confirmation that Linode was the vector.
While there is some relief in finally determining what we believe to be the vector of our attack, it's very unfortunate that Linode engineers are dealing with the fallout right now.
so you're saying you wrote the code that was responsible for the breach(es)? It also sounds like you knew about these issues before you left Linode but didn't point them out while you were there?
Where are you reading this? When he says he was writing ColdFusion, he's referring to the framework/programming language. He's not saying he wrote every single line of Linode's management interface.
Isn't this like the third Linode hack? Back in 2012 one of their cust tools was compromised in order to steal bitcoins. In 2013, cc# and password hashes were compromised, Linode denied it until the hackers showed proof, then did a piss poor job of handling it afterwards.
Why were people still using Linode after their poor handling in the 2013 hack?
To clarify, we did not leave the users enabled in the Linode Manager. When you delete a Linode user, Linode shadow deletes them in the DB by setting an inactive date on the row.
The honeypot user would not have been able to access the account had the credentials been valid, but based on the information given by Linode we did see someone attempt to log in as that user only once around the time of the compromise.
I'm curious what kind of losses you've seen since your own data breach. I'm sorry but you have far too many plausible ulterior motives for taking the position that you are taking.
1) As someone else pointed out, you're an ex-employee of Linode. You went out of your way to hide this fact. I'll refrain from listing all of the very obvious reasons why your word on Linode should be taken with a grain of salt at the very least.
2) Being able to blame Linode for your own data breach is a fantastically easy (although lazy) way to pacify customers about the fact that their personal data was just pilfered by someone on your watch.
All that being said, what have you presented that can be proven? All that can be proven is that you're an ex Linode employee. Everything else is hot air that we're all meant to take your word. Tons of appeal to authority in your explanations. You keep invoking some mysterious third party "expert" security group who conveniently agrees with everything your own company "discovered." If you were actually confident in your own abilities and that of your team members, there wouldn't be an immediate appeal following every attempted assertion.
Plus, even if you really did hire someone, what company isn't going to just say "yes" and agree to whatever PR campaign their customer is saying while dumping wheelbarrows of cash into their pockets? Frankly I don't believe you, and I find your consistent drumming against Linode to be highly suspicious in the wake of these attacks. You're not involved, are you?
The Linode post you're referring to is just saying that they expired everyone's passwords. That's not admitting anything, especially not admitting anything about a separate incident from a year ago. What lawyer would ever take this and say "okay, you can legally publicly blame linode now?" No lawyer worth his salt. In other words, you're full of shit. Your story if full of holes, tells, and I think you should stop posting so much garbage before you're on the receiving end of a lawsuit or are considered a suspect.
Are you Jesse Nicholson? Is Linode paying you to write this? This reads like a Linode executive wrote it. That's the kind of thing you should disclose. Your only two HN comments are regarding Linode; pretty suspicious. You also commented on their blog post:
TechnikEmpire January 6th, 2016 at 10:28 pm. It's hilarious watching all of these armchair experts criticize Linode for the actions of another.
PagerDuty and WP Engine were both compromised 'inexplicably' during the same timeframe at the same hosting provider. Seems pretty self explanatory. Linode didn't disclose their "security firm" so why should PagerDuty? Linode couldn't explain how accounts were accessed and it isn't the first time! Linode is hacked once a year; it's a feature. They need to get their shit together and stop pretending security is a game.
I enabled two factor on my Linode account recently and saw this worrying copy:
> If you lose your token and get locked out of the Linode Manager, email support@linode.com to regain access to your account.
> Should you need us to disable your Two-Factor Authentication, the following information is required:
> An image of the front and back of the payment card on file, which clearly shows both the last 6 digits and owner of the card.
> An image of the front and back of the matching government-issued photo ID.
There doesn't seem to be a way to say "I have my big boy pants on, don't let anyone in under any circumstances". This is the first 2FA setup I've seen that still allows bypass by contacting support.
EDIT: I also find it odd that you have to manually generate a scratch code, and they don't automatically generate it for you. Again, all of the other 2FA setups I've gone through have done this.
> There doesn't seem to be a way to say "I have my big boy pants on, don't let anyone in under any circumstances". This is the first 2FA setup I've seen that still allows bypass by contacting support.
Sadly, this is quite common, especially with organizations that provide phone support[1].
Agreed - especially having read this earlier (not being able to fix it at the time), still not getting the email (like 6 hours have gone by) and I'm resetting because I should and still no communication from Linode.
Concerning since I had to reset my password and regen my 2fa using only my old password...
There are also a number of reviews on Glassdoor where supposed Linode employees say they were asked by management to lie to customers about security breaches.
That's it. I'm done with Linode. I will be keeping one server on Linode, but my new startup where I need several load balancers, Web servers, DB replication will be hosted elsewhere.
OVH is hard to beat when it comes to raw performance/$. Hardware and network has been quite reliable for me (using SoYouStart and OVH), and their DDoS filter is excellent. I'm not really familiar with their cloud/VPS offering.
That being said, they only have one data center in NA at the moment, and they recently suffered a rather nasty network outage in that data center. Would definitely look into them if your target market is primarily EU, but if you're mostly NA-based, YMMV.
I just don't think the performance is good enough for pure web hosting. Much slower/$ than Linode and DO.
Their uptime and infrastructure is great, but you're totally on your own (e.g. if you're getting DDoS'ed). Also their traffic cost is not as easy to predict than DO/Linode/Vultr.
Blog is down. Maybe they should host it on WPEngine. (:
This sort of thing almost became background noise when I worked there. I can say that things improved somewhat after the HTP compromise, at least for awhile.
Yeah it's not like ColdFusion powered web sites enabling billions of dollars in commerce and what with all other web development tools being free of security vulnerabilities and all.
With Linode's extended DDoS I have not been able to get into the Manager for a couple weeks. I'd really like to cancel my account with them (and they do keep billing) but I don't appear to have any tools short of a chargeback. Anyone else in the same boat?
VPS providers that are typically pitched as Linode Alternatives. Not 1:1 equivalents, but may work depending on your needs.
Vultr, Ramnode, Wable, iWStack
Or, depending on the number of VPS's you have, you might like:
- Aliyun, the cloud service from Alibaba
- A dedicated server from OVH, or their mid-tier brand, SoYouStart. This is my personal favorite. They have real DDOS protection, Data centers in North America and Europe, reasonable web interfaces, lots of available IPV4 space, and DIRT CHEAP prices. Run proxmox as the distro, and you get a decent interface to create and manage VPS instances.
The most important piece would be to try and split instances across at least two of these providers so that you have some fast recourse if something goes wrong. For the things I'm running, doing a nightly rsync of the data from one provider to another suffices as reasonable insurance.
This is a perfect poster child for decentralized authentication. Centralized stores of passwords and secrets are just asking for trouble. When will we learn?
so... I completely agree that shared secret authentication is a bad idea, and I use public key authentication wherever I can (password auth is disabled for ssh on every server I control; I do everything with ssh public keys.)
However, I've yet to set a public key authentication scheme that users would find acceptable for web applications. Do you really expect all users to setup x.509 auth in the browser?
What is your public key solution to authenticate the web-applications that customers demand?
OK, I believe you, though I don't know any of the details. It's an important misunderstanding to prevent, though. In this case, including enough of the details to make the comment substantive would probably have been enough to clarify your intent as well.
you could have made the comment more substantive in a way that was clearly not a personal attack, and that would have solved the problem.
This is Yet Another Reminder to use unique, unguessable, unmemorable passwords for all online services. It's a question of when, not if, any particular password database will be compromised. While 'password1' and 'this is my long unguessable password' and 'm4r1g0ld' and 'false lemur capacitor paperclip' will all eventually be guessed, 'sF0PSQMwK85fe9xanqJRm9nty9cJGHJsVmti' will never, ever be.
Why was this downvoted? It's true. The example password is 36 characters; a mixed-case-and-numbers alphabet has just over 6 bit per character; the example password is thus 217 bits. It's not going to be guessed.
Except passwords aren't even a relevant part of this breach. They're worth absolutely nothing compared to the ridiculous amounts of customer data linode controls.
