"Basically, we were in a race to develop new anti-fraud techniques and they were in a race to develop new ways to steal money. The by-product of it was all our competitors got wiped out because as the Russian mobsters got better and better, they got better and better at destroying all of our competitors." - Max Levchin on the history of Paypal.
Those who don't learn from history are doomed to repeat it.
"it was basically became clear that we either figure out how to beat the fraudsters or the fraudsters will take us under. And the company more or less refocused itself as a research entity towards figuring out innovative technological ways of destroying fraud in the Internet. And that alone could be the subject of an entire class or a series of classes so I will completely skip over all the cool technology we developed. Some of you might have seen stuff in the news, words like EOR or ELIA, all these tools that we've built. They're as cool as they sound. I could never tell you about them because they're very secret and they're still in use. But maybe if you want to hang out afterwards, I can tell you a little bit. But they're really cool and we did really figured out how to kill fraud.
...
The submerged part of PayPal is this massive and very, very numerically-driven risk management system which allows us to instantaneously tell when you're moving money to someone else, with a very high degree of certainty whether the money you're moving is yours or you got it illegally and we might be on the hook later on to help the authorities investigate or retrieve the money, et cetera, et cetera."
I simply must know. I'll never find out how Paypal killed fraud unless I ask HN right now, and I've got to know. So, with apologies for branching into a completely unrelated topic:
What magic did Paypal invent in order to do all of those amazing things described above? How'd they kill fraud? What are the details of how the technology works? Is it still a closely guarded secret? Do they use some advanced statistical mathematics to detect fraud, or do they bruteforce the detection by providing massive quantities of data to the otherwise-dumb detector algorithm?
What fascinates me is that they've come up with a way to detect fraud automatically, with no human input. There are false positives, which everyone here has probably had some terrible experiences with, but... Still, how did they train a computer to detect and freeze the very sophisticated and very subtle covert techniques of Russian mobster money launderers?
Any info would be very much appreciated, especially citations / references for further reading on this topic.
I didn't work in risk analysis at Paypal, but I have a passing interest in the field. A lot of it is much simpler than you think it is.
Want a not-so-secret anti-fraud technique? If the billing address is Kansas and the IP is China, you probably shouldn't let that transaction go through. Really obnoxious for those of us who live overseas. Stupendously valuable, though.
There exists a particular anti-fraud heuristic at a YC company. They shared it to me in confidence, because as soon as you know it exists, you can trivially avoid it. I mean trivially. It's apparently insanely effective, though, half because it has a really good handle on who it wants to frustrate and the other half because it's not in the literature at all and, as a consequence, the bad guys don't even know they have to avoid defenses in that class of algorithms. ("But that's security through obscurity!" Good recitation of the dogma, but can I point out to you "This was implemented, in actual computer code, and does in fact actually work?")
There exist more complicated things you can do with machine learning. There also exist more complicated things you can do with heuristics. There also exist more complicated things you can do with live fraud teams. There is non-zero value to finding fraud even after it has happened, because shutting down the fraudulent accounts means you can either keep some horses in the barn or even, potentially, call some of the stolen ones back.
An underappreciated angle of this is that you don't have to outrun the tiger, you just have to outrun your friend. You start getting dedicated adversarial interest as you approach the most lucrative weak link in the entire financial system. That was Paypal back in the day -- with a bullet. Even though no startup/bank/etc ships with perfect security, investing sufficiently in security means the guy that gets victimized is someone other than you. They either go bankrupt or have their problems burned away in cleansing fire. Then the cycle repeats. (One reason Bitcoin companies keep getting looted is that if you plot out "Amount we could conveniently steal" versus "Resources spent on defense in the last 6 months" for all companies in the financial sector there are a lot of dots representing Bitcoin companies which are isolated islands, and the sea is filled with sharks.)
It's a fun field, for a certain perverse and high-stress definition of "fun." I'd probably have made a career in it, but found making and selling software (with just a wee bit of risk management/security/etc thrown in) to be more fulfilling.
There exists a particular anti-fraud heuristic at a YC company. They shared it to me in confidence, because as soon as you know it exists, you can trivially avoid it. I mean trivially. It's apparently insanely effective, though, half because it has a really good handle on who it wants to frustrate and the other half because it's not in the literature at all and, as a consequence, the bad guys don't even know they have to avoid defenses in that class of algorithms.
That moment when I realize I should've come up with a way of demonstrating my own ability to keep other peoples' secrets, so that I can be let in on this one and satisfy this burning desire to figure out what the heck that particular fraud detection technique could possibly be...
Well, clearly I'm going to be thinking all day about fraud detection methods and about guessing your technique. Hmm. I may as well guess your technique openly:
Your technique must be something unexpected, and seemingly unrelated to "actions that fraudsters would normally be careful not to give themselves away with." So I'm going to guess that the technique is to analyze the linguistic traits of each user's password. Most people reuse passwords, and if the fraudsters share a similar cultural background then the password has a decent chance of giving away the fact that a particular user is from that same cultural background, e.g. if the password contains a foreign dictionary word in their native tongue yet their their purported home address is in a completely different country. That sort of thing.
Or if the same group of fraudsters keep reusing the same passwords, or even parts of their previous passwords, then that would probably be enough to accurately detect them. Most people who don't use a password manager tend to reuse substrings of their prior passwords, so they end up leaving a pretty distinctive identity "footprint" by the type of passwords they choose. Their passwords are likely to share a similar structure, such as always consisting of two words followed by two numbers, for example, or always starting their passwords with two symbols like %#.
And of course, if the fraudsters became aware that their password was giving themselves away, they'd trivially dodge the detector in the future.
Hmm. I'll just have to invent my own fraud detection techniques. This is a lot of fun.
> There exists a particular anti-fraud heuristic at a YC company. They shared it to me in confidence, because as soon as you know it exists, you can trivially avoid it. I mean trivially. It's apparently insanely effective, though, half because it has a really good handle on who it wants to frustrate and the other half because it's not in the literature at all and, as a consequence, the bad guys don't even know they have to avoid defenses in that class of algorithms.
Please don't tell me its capitalization of the cardholder name...
On a different note, I'm happy to disclose the one heuristic BCC has since I'm the only one who can get negatively affected by it. I have several thousand names of people who have previously bought BCC, and they are a pretty diverse sampling of a slice of the American experience. If you hit the trifecta of a) a first name I've never seen before, b) you use a free email provider and c) you've never actually made a bingo card yet, your purchase causes my phone to light up so that I can refund it prior to the actual cardholder hitting me with a chargeback. I also add it to a spreadsheet that I periodically bug Paypal's fraud team with.
This heuristic has literally perfect detection and recall [+] against a particular carder ring, which was using BCC to test cards (via our Paypal account, sadly, so I have limited pre-charge options to fix it) this summer.
Paypal, to their credit, has apparently shut down this carder ring, since I can't remember that heuristic firing in 2014. (Edit: Poor phrasing. The heuristic not firing doesn't mean I'm safe. Having a historical level of chargebacks, which is less than one a quarter, rather than having the carder ring induced "20+ in a week." suggests that I'm safe.)
[+] Edit: It's 2 AM and I can't remember if this is the right circumstance for that jargon. What I'm trying to say is, of the universe of charges that I can classify as good or bad at N months after the charge, 100% of the ones this heuristic flags are in fact caused by a carder ring and 100% of the ones this heuristic fails to flag are, to the limit of my current understanding, not caused by that carder ring.
Very interesting heuristic! I can see why data mining could be so useful for e.g. Paypal: analyzing data on known-fraudulent and known-valid transactions, looking for unexpected correlations...
Although in this particular case I wonder if the most benefit comes from simply flagging people who buy the product without trying it... and that is a heuristic that almost any SAAS could trivially implement.
You mean you just see if the first name is not already represented in BCC's user database? Under what conditions are your users required to give you their first name? For the trial or just for the purchase?
Paypal passes a first and last name with purchases, and some customers provide theirs in Settings (it's not a requirement), so I have about 5,000 or so lying around.
You're missing the point. He refunds a purchase if he's not seen the name before plus the other two conditions. My question is: what exactly does he mean by "I haven't seen the name before"?
>>Want a not-so-secret anti-fraud technique? If the billing address is Kansas and the IP is China, you probably shouldn't let that transaction go through. Really obnoxious for those of us who live overseas. Stupendously valuable, though.
It's also a great way to completely fuck over foreign students who use their parents' credit cards to be able to do things like buy textbooks, pay tuition, and so on.
Two factor auth (text/push notification) would alleviate a lot of this pain. American Express is moving in this direction with real-time fraud push notifications/transaction acceptance.
>Fraud management is a knowledge-intensive activity. The main AI techniques used for fraud management include:
>Data mining to classify, cluster, and segment the data and automatically find associations and rules in the data that may signify interesting patterns, including those related to fraud.
>Expert systems to encode expertise for detecting fraud in the form of rules.
>Pattern recognition to detect approximate classes, clusters, or patterns of suspicious behavior either automatically (unsupervised) or to match given inputs.
>Machine learning techniques to automatically identify characteristics of fraud.
>Neural networks that can learn suspicious patterns from samples and used later to detect them.
>Other techniques such as link analysis, Bayesian networks, decision theory, land sequence matching are also used for fraud detection.[3]
It is also used by insurance companies to detect fraudulent claims. From what I understand, the software is very very good at flagging possible fraud and there is probably a human reviewer of the flagged claims.
Paypal wants to keep it a secret because if the fraudsters known what Paypal is looking for, they can change their behavior to bypass the fraud detection. Also, it is a threat to their business if competitors use it as well.
The best part was that they offered them a credit on their account as compensation. As if they would want to continue their business with a hosting company that allows unauthenticated users to issue commands to their technicians.
This is Rogers you're talking about. The sell you a shitty expensive connection capped at 60GB, and then they hijack your DNS failures to resolve to their shitty ads page, and throttle your traffic depending on the protocol. If you drive down the street in my city you see wireless access points named "Rogers Sucks Ass".
>Not only that, they gave an unauthenticated employee direct access.
Now, this is kind of an interesting thing. From what I've seen, lower end datacenters, for example, he.net, often have "free remote hands" - their techs are expected to access your servers. If you go higher end to coresite or equinix? the only people in the building are security. (and /maybe/ a power/hvac person) And while they can get into your stuff (the locks on the racks are a joke) they aren't expected to do so; a security person will get fired for jimmying with your stuff.
As an aside, from what I've seen, he.net has pretty good access control; they check your ID, and then they take a picture of you, and the login system shows the person who is letting you in a picture of you from the last three times you entered. I don't know how strongly they authenticate remote hands requests. He.net also prevents "tailgating" more effectively than most of the other area datacenters.
It's interesting, even within coresite, all the local coresite locations have a fingerprint reader + radio access card access control combo, but at 55 S. market? they don't even try to stop tailgaters, as long as one of you has a badge. at 2972 stender? they put some effort into it, though they aren't as effective as he.net.
I think that he.net is more effective at preventing tailgaters precisely because the user doesn't get an access card. You have to ask one of the he.net security folks to open the door for you every time you leave the datacenter floor, which presents a reasonably non-confrontational way for them to verify everyone has been verified.
At 2972 stender, they are very apologetic, but they do usually stop you. I have, however, seen people get through without verification during busy times. (the 'man-trap' system is imperfectly enforced, in part because it would clog the place up otherwise.)
Of course, these comments are mostly about physical security, and the article was about getting unauthorized remote hands. I didn't use the remote hands much at all at he.net, so I don't know.
Before people start commenting on how bitcoin needs this or that or predict the doom of bitcoin, read the article: this has nothing to do with bitcoin's security. The attack was perpetrated in the most inane social engineering way, and actually, the bitcoin exchange was smart enough to not put all of its bitcoins in the same basket, so 100,000 CAD worth of bitcoin is not a death blow to this exchange.
Most of real world fraud is exactly this kind of stuff. Bitcoin solves a problem which is not actually a problem in real world. And leaves the problems which actually are problems unsolved.
Don't get me wrong. The proof of work ledger is a cool idea. However using it as a currency in this kind of implementation with complete disregard to anything else doesn't work in the long term.
>Bitcoin solves a problem which is not actually a problem in real world.
Well, it solves more than one problem, and a few of them are certainly real-world.
>However using it as a currency in this kind of implementation with complete disregard to anything else doesn't work in the long term.
Would you care to expand on what you mean by "complete disregard to anything else"? I mean, clearly this is hyperbole, but I'm curious what you are referencing specifically.
I simply meant that the bitcoin as a protocol and the standard client implementation right now pretty much ignores everything else. Some can be fixed on the client but it is not there yet. As an example the wallet is unencrypted by default.
In addition one of the biggest problems with bitcoin stems from the problem it solves. Transactions cannot be reversed, thus making thefts extremely lucrative.
Normal banks simply reverse transactions if they are fraudulent. As an example someone hacked Nordea (Bank operating in Nordic countries) last spring and got away with 600 000 euros. All of the transactions except one were reverted and the missing one was compensated by the bank.
Rephrased: it is a feature, but it implies a tradeoff: people who are considered more important than you control your money. One which I don't consider worth it. I've never had a problem with stolen assets of any kind other than in the physical world, and I consider my physical assets that were stolen fair, I would not want a God government who can see every single thing that happens in any place and any time so they can recover my physical assets.
The decentralised check against double-spending is bitcoin's only solution. And it's quite a genius solution, in a way. Bitcoin is not the first cryptocurrency, but it's the first decentralised one.
Other than that, bitcoin behaves more like cash than like banks or credit cards. You can build banks and credit cards around bitcoin, and you should, but bitcoin itself is just like cash, and that's all it should be. It doesn't need to solve other problems that as you say have already been solved by systems like two-factor authentication and the like.
This particular attack is more like thieves walking into a bank and claiming to be a person and then being able to get cash from the bank's safe (an analogy that also fits nicely with the relatively small amount of money they were able to steal). It is not like being able to rack up fraudulent charges on a credit card.
I only have a vague memory/understanding the subject, but I believe the Bitcoin protocol supports the concept of multi-signature contracts. That could be used to create decentralized escrow/trading/exchanges, which means that you would never have to let a single entity control your private keys. (https://en.bitcoin.it/wiki/Contracts)
I don't think there is a working implementation of the multisig contracts in the wild yet, but the protocol does support a way to solve these problems.
That won't work for traders, transactions on the blockchain are too slow to do what exchanges really do. Traders would not use such an exchange, it'd be too slow.
> Most of real world fraud is exactly this kind of stuff.
So what? Banks have the same problem. If I call up my bank and ask them to do something, they'll ask for a bunch of random information I didn't know I'm supposed to keep secret.
> Bitcoin solves a problem which is not actually a problem in real world.
What? It solves the problem where some incompetent fucks are in charge of your money and screw you over, which is exactly what happened to this company (except they trusted them with hosting instead of money)
Surely the fact that the attack was so inane does comment on bitcoin's security. OK, not the inherent security of the basic protocol, but the actual real-world security of real-world implementations.
You'd never store $100,000 actual dollars in a place where the employees let anyone walk in and take it if they just claim to be you. Bitcoins are equivalent to cash but they are not being stored anything like how cash is stored. It's a fundamentally difficult problem that bitcoin requires you to solve, so I think this really is highly related to bitcoin's security.
Although bitcoins are more like cash than like credit cards, their ethereal nature allows you to lock them up in ways that cash that cannot be: encrypt them (i.e. you encrypt the files that contain the private keys).
This is not a solution that bitcoin itself needs to build into the protocol. It's just something that we already have that should be used when using bitcoin. People are taking a while to get used to the idea of securing a decentralised currency.
They could have kept the hot wallet req multisig to move coins, then backed it up somewhere else so even deletion wouldn't be a problem. It's all the low hanging fruit getting fleeced Bitcoin protocol offers plenty of safeguards if you use it
This has everything to do with Bitcoin. Yes, the attack vector is old and boring, but it's a proven one that has been used more or less the same way to hack multiple Bitcoin exchanges.
Bitcoin transactions are irreversible, so there is no way to get back stolen coins. And there is insufficient consensus to block them from being spent in future. These, combined with the magical pure-virtual, ethereal nature of Bitcoin, makes it an unprecedented target for thieves.
Maybe Bitcoin can yet be extricated from these issues, but the basic system Nakamoto gave us does not address the needs we so clearly have. Given the track record of so many exchanges and other services, we have a long, long way to go.
Most people who read HN probably would never keep more than a thousand bucks or so in cash. Once you get a bunch of it, cash starts to feel "unsafe" and you deposit it at the bank, whereupon it becomes an obligation, a sort of "virtual currency" if you will. This "virtual currency" has various ways of being transmitted, and each way is at least somewhat reversible.
What about Bitcoin? If I have a bunch of it, can I convert it to an obligation with a reputable counterparty? One who will not store it in a form which makes it so easy to get away with, once stolen?
We can treat it like cash, but then we must wonder what we are doing piling up cash in our exchanges. Conventional exchanges do not work like this at all--there might be a good reason for that.
You bring up a good point, in how most dollars are "virtual". And indeed, we have seen the same kind of "virtual" bitcoins, where exchanges traded bitcoins that didn't exist in the blockchain (Mt Gox did this for a while).
Again, this sort of "virtual" currency (more technically, M2 currency), is something that you can build on top of bitcoin, and something I don't think that the bitcoin protocol itself needs to build, since we already have it.
You shouldn't keep any third-party balance of bitcoin. You can do any transaction you want from your own wallet, so there's no point putting them in a "bank." Currently, exchanges require keeping a balance, but that's not a safe thing to do. Hopefully someone will make an exchange that makes sense.
And people generally do not do business in cash. In fact, business done by giving someone a suitcase full of bank notes are usually of questionable nature.
In the first world, yes, but this is common in countries with a less developed financial system.
In China, for example, bank transfers aren't great and it's pretty common to buy things like cars with cash. It's especially crazy in China since the largest banknote there is 100RMB, roughly equivalent to 16USD, but cars cost about twice as much as they cost in the US, so it requires a lot of cash.
But if I get robbed, I can contact the police and there are consequences if the thieves are caught. If my car gets stolen I have insurance I can claim off. There's no such thing with Bitcoin.
Why not? As far as I know, the law will treat your bitcoins like any other property you own. It's not legal to steal them just because they're "bitcoins", nor are they inherently uninsurable.
Now, whether that's an effective deterrent and whether there's a realistic chance of catching the thieves is a big question, but there's nothing inherently different. It's not that hard to rob your house of cash and not get caught either. The main reason it doesn't happen is because the criminals who are good enough to do it have bigger fish to fry. If they found out that you had $100,000 under your mattress then there would be trouble.
Neither of those things are impossible to build around bitcoins. If you catch the thieves, you can probably get the bitcoins back. The currency used has nothing to do with whether it's possible to build insurance around it.
It's not even clear what you're complaining about here. Exchanges get hacked because they suck. Banks don't affect the customer as often because they are old and already have been hacked many times before, and have all kinds of insurance and hand wavy ways to avoid liability when they actually get hacked.
It's as if you are discontent that you are now liable for walking across the road safely after being driven around by a private taxi all your life (if the taxi crashed, the company would not lose any reputation, it would be the driver's fault, despite the low wages paid to the driver, pushed to the limit time pressures on the drivers, the recent line of flawed cars they ordered and decided to keep to expenses, and you would die, and it would just be like any other day).
Ouch, there is a reason banks usually have leases that don't give the landlord any access to their buildings. If you're hosting a system that has more than a few thousand dollars of nominally liquid assets around you really really have to start with a secure computing environment. That means locked buildingings where only you have the key, and security audits (cameras, key cards, biometrics, as much as you can get) and ideally additional insurance provided so you can charge failures on the part of the colo against their insurance up to the amount of assets you keep at risk.
What is the current ratio of exchangees that lost customer money to ones that haven't? It feels like it is close to 100%. That can be very damaging to the long term success of btc.
In my opinion the two main problems with Bitcon are the complexity and the libertarian philosophy around it.
Currently the first problem is being exploited. People struggle with how to keep a bunch of long numbers safe and wrestle with vague abstractions - hot wallet, cold wallet. People should know better, but they don't, why? Because it's complicated.
People will probably keep losing money for a while because of that, until everyone gets a handle on things. But then there's a much bigger unsolved question though: How will the libertarian thing work out?
In Bitcoin threads the phrase "IF A MAJORITY OF THE MINERS [decides to screw up everyone]" appears occasionally, usually tempered by "...but being all rational, they won't!"
I predict that there will be at least one serious attempt at that during the next 12 months. If the first boom has subsided a bit, and fewer new people seem start with bitcoin, it would be time for a collusion of people who are able and willing to try out that angle.
And please take "predict" not as gloating, but rather as a warning. Are you sure you understand bitcoin well? If not, why not get out now?
> two main problems with Bitcon are the complexity
Agreed, we need to build better software around the bitcoin protocol, with a cleaner UI. People are working on this problem. (haha, "bitcon", I bet that's gonna turn into an insulting dysphemism soon).
> and the libertarian philosophy around it.
There isn't any inherent philosophy in the protocol. You can use bitcoins with a tinfoil hat firmly planted on your head, or you can use bitcoins to build centralised banking again if you want to. It's just a tool.
It does not say "Libertarian" on the label, but in Bitcoin debates there's always the ideas of doing it without a government, without a central authority, of having competing currencies and exchanges, of "unprotected" economic activity, of "one dollar one vote" rather than "one person one vote", a gold-rush mentality built-in (rather than "let's make a new currency and everyone starts with 100 eBucks").
So it looks like a nice real life Libertarian experiment to me.
>>There isn't any inherent philosophy in the protocol. You can use bitcoins with a tinfoil hat firmly planted on your head, or you can use bitcoins to build centralised banking again if you want to. It's just a tool.
It's a deflationary currency that enables decentralized commerce. I think that in and of itself speaks volumes about the philosophy behind it.
It's trivially easy to create an altcoin that inflates to infinity instead. It's just changing a relatively small amount of code. The difference is instead of a political elite that only cares about themselves & their tribe like most human beings, the monetary policy is set at the start and is completely predictable.
Right now bitcoin is the star of the ball, but the other altcoins represent a significant amount of value. There is nothing guaranteeing that bitcoin will stay the star.
I don't understand enough about the economics to understand what kind of philosophy a deflationary currency entails. Can you please explain?
As to the decentralised nature, I'm not convinced that really appeals to any deep philosophy other than our greedy nature to want to be able to own what we think we should own.
Essentially, a deflationary and decentralized currency encourages "our greedy nature" rather than discouraging it. Inflation is often referred to as a subtle tax, because it devalues currency. Thus, you're incentivized to spend your money rather than save it. Deflation, as the inverse, increases value of currency and incentivizes you to save your money rather than spend it.
There's nothing particularly deep about libertarian philosophy. (Though there are different amount of nuance depending on which flavor of libertarian you manage to get into a conversation with. Some of them are competent thinkers; I have found them to be exceedingly rare.)
Many flavors extol the virtues of selfishness, though. You'll find this is particularly true with libertarians who agree with Ayn Rand. Even those who don't will generally say that people should look to their self-interest (it no longer appears popular to invoke the invisible hand at this point; maybe because of the intervening years of people noticing that Adam Smith had a lot to say that contradicts libertarian ideas) and use abstract forces like the price mechanism to determine policy. (This is a very rough sketch.)
...and anyways, I'm spending too much time trying to be fair. Libertarianism is fucking stupid, which is an opinion I have based more on its political bits, which is a teenager's idea of society, than on its economic bits, which I'm willing to concede more because I'd rather claim ignorance than because I think there's any merit there.
The point is that looking out for number one is a cornerstone of varying intensity in libertarianism and Bitcoin's deflationary and decentralized nature plays very directly into that.
b) some tech support dude who just wants the boring work day to end so he can go play video games who doesn't know anything about you accidentally letting an imposter act on your behalf because the corporation uses subjective (and undocumented / secret) methods to verify their client's identity
> Are you sure you understand bitcoin well? If not, why not get out now?
This is precisely the reason I can't wait to get out of banks and health care. Bitcoin is a cakewalk.
This doesn't absolve Rogers of responsibility for this, but the incident occurred right as the data center was being acquired. Depending on how far along the transition was (per [1] it doesn't seem like it would have been very far) it could be that this was an issue with Granite Networks' security protocols and not Rogers'.
Could also be the exact opposite. It's mildly suspicious that this happened so soon after the acquisition. It's probably a coincidence, but it smells at least a little bit like Rogers hooked up their own support system without proper training.
Everything seems like obviously Rogers' fault up until this:
"“It’s completely ridiculous,” said Grant. “All they did was go on the chat session and say, ‘Hi, I’m James Grant and I have a server with you’ and the data centre said, ‘Yes you do, what can we do for you?"
If Rogers is like many ISPs, where you can chat with a representative but only after you've logged into their online portal, this seems to suggest that the attacker had already passed through authentication into Grant's account on Rogers, which is why they did not do additional verification. I agree that additional verification, in a form not required in the initial login, should be done at the beginning of the chat, before taking any user-requested action. But the article doesn't specify how the hacker in question obtained access to Grant's Rogers account to begin with. Until we know that we can't fully ascertain the extent of Rogers' liability here.
Don't most companies let you do a live chat without logging in first? One of the things you might want to handle with a live chat with tech support is "I can't log in", so it would be odd to restrict it to logged in users.
For this particular situation, I just went to rogers.com, "Contact Us", and followed the links to a live chat. Never asked me to log in, although it did ask me for contact info and I stopped at that point.
I don't know if that gets me access to the servers, but once you're in a company in any capacity you may be able to socially engineer yourself pretty far.
Given how much physical security the article says they have, I seriously doubt that Rogers would take action on any server through the anonymous, public-facing live chat. It is possible, of course, but I wish the article would clear that up.
Keeping 194 BTC in a hot wallet on your server is essentially the same as trusting $100,000 in cash to the minimum wage security guard at your bank. You'd certainly hope that the bank has procedures that he's going to follow correctly, but $100,000 is a lot to risk so perhaps a little extra security would be a good idea.
Mind you, you'd also rightly expect the bank to cover the loss if the guard just hands it over to a stranger.
I don't know the size of the exchange, but it's reasonable that they would have enough bitcoins in the hot wallet to process withdrawals on an "unlucky" day - that is a day where significantly more people withdraw than deposit. That could easily be 150BTC for even a small exchange. (the article says 150 were stolen, not 194)
considering going up a level in security basically means hosting your own servers, it's probably worth the risk as long as you assume your hosting provider doesn't give strangers root access to your box
A while ago I wrote that perhaps the greatest contribution the Bitcoin experiment will make to humankind is to teach you and me and our neighbors more about the realities of economics. And later I added that the Bitcoin experiment will also contribute to greater understanding of attack surfaces and online crime. Many of the ideas about how to mine Bitcoins, store Bitcoins, and trade with Bitcoins as a medium of exchange illustrate both the strengths and weaknesses of any other medium of exchange in a world full of human beings. Seeing the discussion of Bitcoins here on Hacker News reminds me of early online discussions in the 1990s of online payment systems such as PayPal, and the arguments beforehand that PayPal wouldn't have to invest a lot of time and effort (as it eventually did) building defenses against theft and fraud. If a weakness in a system is attached to a lot of money, the way to bet is to bet that someone will go looking for that weakness, even if you haven't thought of it.
“Rogers Data Centres provides the highest level of security in the Canadian data centre industry. Its security protocol is operationally certified and in accordance with industry best practices. We have reviewed our security processes and continue to work with our customers to make sure they take advantage of all of our security features.”
It looks like it's time to use full disc encryption even in datacenters now. (there are scripts available to startup ssh and allow remote password input before mounting root)
The problem is that it does require manual intervention on reboot and may not be very useful for 100s of machines.
I feel like that has a very limited scope of usefulness. Datacenter machines are on 99.9% of the time, and generally speaking, thieves don't go through the trouble of stealing hard drives from datacenters.
In this case the wallet was stolen from the machine after it was rebooted into single user mode. With FDE the machine wouldn't have been usable when it was rebooted so the attack wouldn't have worked.
Did you read the article? The server was rebooted into rescue mode and data has been read directly off the disk. You don't need to steal the drive to execute that.
Do you need FDE for that? It would be feasible to encrypt just the BTC keys; the server can reboot automatically but needs repeated authorization in order to access the BTC.
I believe only FDE would help in that case. Otherwise anyone having access to the rescue mode could just replace the binaries on the server to send you the key (and all the bitcoins) after it was used.
Actually even right now I hope the server has been rebuilt from scratch along with grub and everything. If they just restored the server as it was left, for all we know they could be running a rootkit or a hacked version of bitcoin daemon.
Data centers offer many critical advantages (redundancy of power, argon fire suppression, networking and cooling), but I question the notion that they offer greater physical security than, for instance, an in-office server rack.
They have gates and retina scans and cameras, but they have these things specifically because many strange and mysterious people are allowed in very close proximity to your servers: It is a compensation for the greater risk that the environment naturally presents, rather than being above and beyond the normal security.
But it does nothing to defend the physical security when security is theater (which it almost always is, and where social engineering easily circumvents it), or when employees of the data center are a potential threat, as are the many other customers of the data center are the threat.
People can get on aircraft and through international borders with assumed identities. They can certainly get into a data center, especially given that customer service always prioritizes convenience ("Something must be wrong with your retina scanner. Anyways I really have to get to that server now". In most situations, you'll be getting in).
Those who don't learn from history are doomed to repeat it.