The problem with a "warm wallet" is that the hot wallet of 100Btc might need refilling every hour and so if the attacker just sits draining the hot wallet, the admins might keep authorising refills letting the attackers take large amounts before the hole is discovered. If it must be done that frequently then admins won't bother to audit the hot wallet every time they refill it from the warm wallet, which makes the warm wallet a hot wallet.
If hot wallet refills are rare, the hot wallet must be sizable, but then the refills will be sizable too, so an attacker could steal hot wallet (sizable) and a refill (sizable) at least before discovery.
Each step adds an effective rate limiting step and a chance to catch the hole, but requiring manual intervention too often and the required auditing and manual checking will not be done. But any automation just pushes something from warm to hot.
Even with automatic transfers, I think there's a difference between wallets that are queued up and actively sending money out of the system, and wallets that are queued up to do so (or to refill that wallet). The difference lies in the fact that they do serve as an automated rate limiting point, and have different methods for the hackers to extract out funds from them.
An example: if you have one hot wallet with the keys loaded in to the system, and a secondary warm wallet (basically, a wallet that will be swapped in as the hot wallet when the other runs out), you can secure the details of the warm wallet better, meaning that certain read access vulnerabilities will only be able to target the hot wallet. Of course, an attacker can empty the hot wallet, and then the details of the warm wallet will be exposed when it is loaded because of the same vulnerability, but this forces the system to have a chance to validate the current state of the world before the second wallet becomes vulnerable, which is not the case if both were being used as loaded hot wallets.
There are meaningful automated sanity checks and rate limiting tactics that can be used with automated warm/hot wallets.
Of course, if you have a $5mil hot wallet, you should probably hire a "bitcoin banker" or "bitcoin teller" to sit at a workstation and manually deal with some of these kinds of swaps and oversee the audits. Even if you don't immediately patch all security holes, you'll have a much better idea of where you're leaking.
The hot wallet can be refilled every hour. If you are trying to withdraw <1 BTC but hot wallet is empty it goes to warm wallet to be accepted manually.
Yes it's inconvenient. But between inconvenience and losing $5,000,000
If attacker steals X from hot wallet the admins may detect suspicious activity (when hot wallet is drained entirely they must reaudit everything). Even if they don't notice anything the attacker gets 2X. While warm wallet is something like 100X
The "manually adds funds" flowchart block looks, to a banker, like a risk item. How does that work? Every hour or so, some clerk opens a safe full of USB sticks and takes one out? Now you have an insider theft risk. Did they copy one? Who has physical access? If you're operating 24/7, more than one person will need access.
A secure off-line "cold wallet" system is hard. Physically, it looks like a collection of storage media. Those devices need to be backed up. But backing them up means making a copy of the private keys, which means the backup medium and the backup machine have a path into the funds. How do you audit a cold-wallet device, short of sending the funds in it back to yourself? Looking at it visually is useless, and reading it out exposes the keys to risk.
Some of these problems have known solutions from bank currency management operations. But some don't. In currency management, you can count and recount bills. You don't have to worry whether some of them have been used and invalidated. Bitcoins in some stored form are not like that. Just because you have a sealed plastic bag with a USB stick and a label that says "1000 BTC, validated on 2013-04-21" doesn't mean it's still good. If there was a copy made at some point, someone else may have the keys to those BTC. Even if they haven't transferred them yet.
So manual handling isn't a magic bullet. Off-line vault operations for a sizable Bitcoin operation are going to be complicated and expensive to do securely, more complicated than bank cash vault operations.
We've already seen what happens when they're not done securely.
> The "manually adds funds" flowchart block looks, to a banker, like a risk item. How does that work? Every hour or so, some clerk opens a safe full of USB sticks and takes one out? Now you have an insider theft risk. Did they copy one? Who has physical access? If you're operating 24/7, more than one person will need access.
Unfortunately, there's no vendor in that space with a track record in security. A major provider of hardware security modules ("hardware wallets") for Bitcoins was Butterfly Labs. Their Bitcoin mining hardware business did not work out well for their customers. (As in, forced by the FTC to shut down and refund customers who paid for hardware and didn't get it.)
There are some little startups selling hardware wallets, or trying to get funding for doing so, but nobody with a solid track record of protecting large sums of money has a product. SafeNet and Thales are absent from the Bitcoin field.
The API with a single call "createWithdrawl" is very good.
Having code like this to begin with changes how all your future code works. It also is easier to debug, simpler to explain, etc. You think twice about adding new API calls, which makes you think twice before you add new bugs. Having the ability to muck around with API calls you shouldn't be making will catch you eventually.
I've posted this before, but web-app security in most startups I look at is so bad. If it's written in PHP, I can normally find every bug in the book in under 5 minutes. What is worse on top of that is many companies don't respond (and don't fix) when you report bugs. I'm talking well known startups too.
I really think the OWASP Top 10 needs to be required reading for startup founders (or technical leads).
Essentially, start making exchangers look more like banks, with manual interventions to evaluate whether transactions are valid.
The bank system (and transaction clearance delays) evolved for good reason. I suspect that bitcoin will not achieve the validity of fiat until a similar set of checks and balances are put in place (at least for sizable transactions.)
Yes, I don't know how it works with banks but here it's essential to loosely couple your services. The main app shouldn't controll the funds/transactions app
Another solution which has been proposed is to separate the role of "money gateway" and "exchange". If an exchange has a provable amount of currency on its books (of every trading on the given platform) and doesn't directly manage bank accounts, solvency "auditing" will be provable, happen in real time and the exchange would fail fast.
I take issue with equating web developers and amateurs.
I understand his point. And I conceed that there is a lower barrier to entry for writing a PHP signup form than a Go systems-level program.
But please don't assume that all web apps are written by 2nd-year junior programmers. Some of us web developers have the experience required to write secure web apps.
I think the point is that most web developers (even very good, experienced ones) are amateurs at topics like server security or financial accounting/controls because those topics are outside of their domain knowledge.
It takes more than not being a "2nd-year junior programmer" to write secure, reliable code; it takes specialist knowledge to make a solidly audit-able financial control system.
Right, I've been programming for a living somewhere over a decade and while I certainly consider myself a professional, that's "enough of a professional to know I don't know nearly enough to write such a thing without expert help".
The post is not about amateur developers, everyone makes mistakes. The point is amateur developers make bitcoin apps a low hanging fruit. And that's why you have to be prepared to be hacked right now.
Also I wanted to recommend manual approach to large withdrawals rather than automatic.
If hot wallet refills are rare, the hot wallet must be sizable, but then the refills will be sizable too, so an attacker could steal hot wallet (sizable) and a refill (sizable) at least before discovery.
Each step adds an effective rate limiting step and a chance to catch the hole, but requiring manual intervention too often and the required auditing and manual checking will not be done. But any automation just pushes something from warm to hot.