Just my opinion but I think the conspiracy theories are not realistic. Coinbase is essentially a startup that is dealing with monstrous traffic. They are still scaling their infrastructure and when the price starts moving quickly everyone logs in to either get in on the action or take a look. This cranks the traffic up and the servers cant handle it.
I am not excusing their behavior as they need to be able to handle it, especially when their server status affects the bank accounts of real people. But I feel that it is understandable with out condoning it. The simplest explanation is likely correct in this case.
I don't think many of us could have predicted the volume and interest in BTC a year ago and neither did Coinbase's purchasing and infrastructure teams.
I've used Coinbase on and off for a while. The service always felt exactly as you describe it: startup busting their collective ass to keep up with demand.
Given the recent increase in news about cryptocurrency I've expected "technical difficulties" with Coinbase. Heck, most mining pools are sweating bullets right now.
I would guess they're also likely the first stop for new entrants, as they're (or were?) the only US based exchange, easiest to use, and insured by FDIC/Lloyd's.
Coinbase has been my recommendation to anyone new to the scene who wants to buy/use bitcoin.
And obviously once finished, move all funds to paper wallet.
Is it really that much traffic compared to your typical consumer-facing start up? There can't be that many people buying BTC compared to the number of people that say, Stripe or Whatsapp handled in their early days...
I don't know about Stripe or Whatsapp, but I know coinbase has had an average of 30,000 new users a day for a while, with spikes of over 100,000 new signups in a single day around thanksgiving.
It's absolutely bonkers amounts of growth!
Edit: more concrete numbers:
In January of 2016 they had around 2.6 million users.
On November 29th they announced they had over 13 million users.
Whatsapp famously had ~450 million users when they were acquired by Facebook. I assume that the average user sent quite a few texts, some pictures, occasional video... they also supported voice calls...
I don't know how much computing power is actually needed when you're just buying BTC. When I used Coinbase recently to take a small BTC position, I only made maybe 4 transactions in total. Is this really a lot to handle?
I don't have any experience in this area, but I imagine that fintech is a bit different than messaging. It's not the amount of data, but the nature of the data.
I'd imagine you can't use scaling tricks like eventual consistency or simple sharding here. Traders are going to be pissed if their most up to date information from the exchange is a second behind what they can see.
It's already at the point where they have a line in their FAQ for the GDAX API that tells people where to colocate from for the lowest latency.
And all of that data needs to be stored damn-near indefinitely, and securely. And integrated with traditional financial systems.
I'd love to see some information on what it actually takes, as i'm just guessing here, but seeing as there aren't any bitcoin exchanges that can seem to keep the servers running smoothly during spikes, i'm guessing it's not a trivial problem to solve.
Thanks, I'm not in finance but it seems obvious in retrospect that not being able to rely on things being eventually consistent is a bottle neck. Before thinking about this, the number of users that people threw around when talking about BTC exchanges seemed very small.
... from 9 to 5, monday through friday, with a fairly predictable load, and decades more experience.
I'm not saying it's impossible, but the amount of growth they have seen is substantial and unprecedented. Nobody could have predicted it, and the rate of increase just keeps going up.
They are clearly new in this space, as all cryptocurrency companies are. I don't think that means that they should just give up and go home because they couldn't keep the servers up during spikes of insane activity...
That, followed by the fact that many of their users probably sit on the site a good part of the day constantly refreshing the page/taking up websocket bandwidth to watch price changes. Yeah, load is a serious issue.
I think its also a matter of funding and company financials as well. You have to make money to justify spending money and each company has a different idea on what an acceptable infrastructure spend is. While it might be great to spend a ton of money and build a bullet proof infrastructure, if traffic slows down or revenue dips they are in the very negative position of having a lot of hardware and not a lot of money in the bank.
They have to scale at the rate their revenue or financial backers will be able to sustain. Based on the number of new users signing up to the service the down time is not yet a significant deterrent. If use drops because of downtime they will likely need to reevaluate a large capital spend to solve the issue.
"We want to run an exchange that the entire world will use for this crypto-currency that's become very popular and has increased in price 1000% since it became widely known."
I think I would have predicted that they'd need to run at a very large scale, and be able to handle massive spikes. Whether or not I could have convinced them to make the investment is another story.
The problem is what happens if that growth stops? What happens if bitcoin tanks right after you hired 300 new customer service reps and tripled your server hardware?
It's a gamble no matter what, but moving too quickly or moving too slowly results in the company going under.
In that case, I think straddling the line of "overwhelmed when there is a big spike" is a good spot to be in all things consitered.
The optics on "servers are down due to a massive spike in volume" are much better than "Coinbase had another round of layoffs today..."
Currently they are signing people up at a dramatic rate despite their server issues. If they start laying people off because they spun up to fast they could start a panic that would kill them.
They have to weigh all of this in when deciding on an infrastructure spend.
I would also like to think from a purely personal level that they don't want to hire and fire people like that as these are real people with real families involved.
Coinbase goes through this charade every time this happens with a major digital currency. It wasn't a legitimate excuse the first time, and it's not an excuse the 7th time.
Scaling is a solved problem. You have AWS, Google Cloud, and MS Azure to choose from with financial institution/HIPAA-level offerings.
Bitcoin doesn't scale with "The Cloud". If anyone can get the private key, they have the money... sending it off to any other provider is going to end badly.
Sure any other company doesn't care if "The Cloud" leaks.. it's just the customers problem, cause it's their data... However Bitcoin leaks, it's the companies money, so it's now again their problem.
The thing about scaling is that you can't justify the engineering involved in overscaling until you're reasonably sure you'll need the capacity. Otherwise that's called overengineering. Managers get fired for approving spending on things like that when the demand does not appear. It's also hard to simulate real traffic because they don't always behave how you predicted they would.
Also, your system gets more and more complex as you scale. More monitoring systems needed, more attack vectors to watch for, deployment systems, backup systems... Imagine the security level coinbase needs to have. It's somewhat unprecedented since a hack could make off with a lot of valuable assets in an instant. (even just from their hot-wallet)
You saying scaling is solved makes me question what you think their infrastructure looks like. It's not just an AWS loadbalancer with some shards behind it. It's A LOT more complex than that.
You really seem to be wringing your hands pretty hard about an overloaded NGINX instance. Wouldn't a conspiracy that was stealing everyone's money be a lot worse?
No I'm saying that any reply approximating "its fine guys, they're new to this and won't anticipate everything" is disquieting when lots of coin is on the line.
No architecture scales infinitely, you merely shift at each step what the next bottleneck will be. Even with small scale sites you'll see this: first, you need to move from everything on one server to a load balancer plus app servers talking to a single database. Then you need to share your database. Then you need to move to multiple load balancers. Then so on and so forth, depending on the specifics of your service, mixed in with various rounds of local optimization at the application level that buy you just a little more time before having to ship another architectural change.
Most distributed systems don't really scale ass you add nodes beyond a factor of 10 to 100. If they did, you wouldn't need to hire super expensive people to handle those systems.
The cloud enables easier scaling, it's not a silver bullet. No matter what, scaling plans have a region where they break down, so it doesn't imply they aren't designed for good scaling.
My friend thinks the exchanges also go down when assets "flash-crash", because it means normal users can get in to sell their assets. And when they can't get in, they also can't cancel their stop-loss orders, so it means a TON of assets are available to buy cheaply (by the exchange and/or certain VIP users/collaborators) as the value goes down. Then these are bought up, and the price quickly increases again, before the exchanges slowly come online again.
Therefore he thinks the people exchanges are either part of the "flash-crashes" (and buy up assets themselves, because they know exactly where most of the stop-loss orders are placed), or they get kickbacks from the whales manipulating the market, by giving them access to the database with all the limit and stop-loss orders.
This has always been a worry in the traditional stock market. With a stop loss order you're basically creating a "secret" order to sell at below market value. Anyone who knows about it has a direct incentive to profit from it.
Stop-losses are very important in investing but I'd recommend always executing them manually at market rate. Set a price alert to notify you. Of course if the exchanges are down you're sol, but if BTC is on a bullet train to $10 your exchange stop-loss probably won't fill at your price anyway.
"Stop-losses are very important in investing but I'd recommend always executing them manually at market rate."
No, no, no, no ...
You do not ever want to insert a market order[1] ... the market can be manipulated and high frequency traders can buy/sell your public market order at an artificially low/high price.
The answer to getting "stopped out" is not to insert a market order - it is to create a fill-or-kill at a specific price or other more specific order type ...
Stop loss means get out NOW to stop losses. Trying to finesse a nickel out of a trade at that point is a sucker's game.
Getting "stopped out" is fine, it's part of investing. You need to stop your losses. I'm just saying don't put the order on the books for the exchange to pluck like a ripe berry.
Coinbase being unavailable during times of extreme volatility is not a new thing. Today's swing was 14k up to 19k then down to 15k or so over a period of an hour of severely degraded access.
I'll go out on a limb and say that much of the recent spike can probably be attributed to the 100k++ accounts open on Thanksgiving weekend getting funded today and everyone losing their shit mashing the Market Buy button which get instantly executed. The mechanics of vast market buy orders with no liquidity is well known.
If there's a very large value swing, either up or down or both, services such as this are guaranteed to have to drink from the godzilla-scale firehose and learn to like it.
If there's a run on the bank, the little people are gonna get screwed, no doubt about it, no matter what store of value we're talking about (seashells, gold, paper currency, cryptocurrency, gold-pressed latinum, etc).
That's what they did to me. Incredible how much naivety surrounds and excuses their behavior. There is no recourse when they "accidentally" lose your purchase at $200 once it jumps to $450.
Bank runs are bad when there isn't a full reserve. Coinbase should carry a full reserve.
Fractional reserve banking is a prisoner's dilemma, we all marginally benefit from the status quo of credit/debit/interest, but in a crisis, the first people to pull out win, and everyone else loses.
They presumably have on hand the full amount of cash and Bitcoins that their participants have given them and don't loan any of them out. That means they have a full reserve.
They only need(* depending on TOS guarantee) to keep the full amount of Bitcoins matching your Coinbase wallet. They can take all the cash they've been given, pile it up and set it on fire if they like.
1 BTC at Coinbase is only worth whatever Fiat currency Coinbase is willing to buy it from you for. If that's $1 for all your Bitcoins, tough luck.
They're under no obligation to guarantee a sale price for BTC.
I wonder how apt the comparison to a bank run is, given that AIUI Coinbase holds the Bitcoin of the exchange participants. Is there any legal form of sequestration of those assets?
Bank runs are bad when there isn't a full reserve. Coinbase says it has 100% reserves.
Bank runs are how fraudulent exchanges are caught: Bitcoin is self-auditing, and either you have it or you don't, there aren't IOUs or paper receipts.
Fractional reserve banking is a prisoner's dilemma, we all marginally benefit from the status quo of credit/debit/interest, but in a crisis, the first people to pull out win, and everyone else loses.
The big question now is, what happens when somebody sells big for dollars. This would be an excellent time for someone with 10,000 Bitcoins to sell, take their $160,000,000, and go home. Could the market handle that?
On GDAX the trade volume for BTC/USD is now about $1.4 billion per day. And there are other exchanges too. So yes the market could fulfill that trade. Would probably bring the BTC price down but it wouldn't be catastrophic.
Given the completely unregulated nature of these exchanges, that 1.4 billion USD could be legit, it could be wash trades, it could be inhouse bots doing crazy things to rig the market. , or it could just be a complete fabrication.
As somebody who doesn't have a background in finance, what are the legal obligations of an exchange to stay online? How can a user of the service be certain that an exchange is offline for technical or non-technical reasons? Let's imagine the likely scenario when the price suddenly drops like a rock and the exchange goes offline completely (for technical or non-technical reason). Could a user make a claim that Coinbase is liable for their losses because they were unable to access their coins?
A financial institution has fiduciary obligations to its clients to maintain the assets (here, the Bitcoin, ether, etc.) entrusted to them.
However, financial institutions are not currently obligated to maintain their clients' ability to transact those assets, let alone within a specific time-frame. Many people have sued their brokerages for not letting them sell stock in a timely fashion to take advantage of price changes; essentially all of them have lost or settled for pennies on the dollar.
In this case, it's completely up in the air - bitcoin isn't an official financial product, doesn't fall under any formal juristiction, etc. At best any gains get taxed now, but besides that, exchanges can't give any guarantees about uptime or e.g. getting your money back if they're not available.
I'm expecting this all to go wrong, up to the point that I'll just let my coins ride it all out. The first gamble is bitcoin, the second gamble is the exchanges.
It's a matter of time until Mt. Gox part 2 I think. It may not even take nefarious workings at the top, just sloppy work somewhere + one bad employee or outsider.
Is this going to end in a re-discovery of exactly why we have FDIC? Or the SEC? How do people even know how much bitcoin activity is real and not essentially faked?
Popular Google Searches by year, by my estimation.
I do not think it would be difficult for someone to setup a website and scam newbies into buying 'bitcoin', handing over credit card info and taking wire transfers. Perhaps even act legit to start and then segue into only processing part of transactions as real. Quite a ponzi. I don't think is a stretch either. There isn't anything comparable that people would give you money for and the key really is the overwhelming interest and publicity combined with people who have no clue what they are buying. As long as (as with any ponzi) people don't withdraw more than they actually have this could go on for quite some time. Madoff had it much much harder.
You can't easily pull a scam like that using credit cards. Wire transfers could be potentially easier as we know the union transfers are hard to revert (but there's work being done there as well).
Essentially, if you interact with banks and have too many chargebacks or complaints, you'll be dropped. Or your fees will get massive. And that's if you find a payment processor who's happy with you taking money for cryptocurrencies in the first place.
It's not about refunding the customers though. It's the payment gateway that will drop your as a customer. Neither them nor Visa/MasterCard want anyone running scams via them.
If I were running a scam, I'm pretty sure at the moment I would have a positive cash flow: a lot of people are transferring their USD/EUR into my bank account because they want to get on the Bitcoin train. If a few people are cashing out, I'm pretty sure I would be able to manage to give them their payouts. Exchanges also have a withdrawal limit of a few K's per day, in my ponzi scheme if someone wanted to withdraw 100K it would take maybe 10 days, and one hopes there are enough new suckers that each day I'm able to pay out, out of my official bank account, my slush fund. The rest of the money can be transferred out where no one can touch them...
And when the whole thing blows up, I disappear. My payment gateway will drop me, but what do I care?
Last week, their rates were about 4 BTC to rent 1 TeraHash/second of mining capacity for 24 hours. Today, according to a mining calculator[0], that works out to about $2.62 in BTC per day.
You really shouldn't be surprised, at this point, by Coinbase going down during times of high activity. It happened on November 29th (when, in fairness, everyone was going down), and happened four times last June, which should have been their canary-in-a-coal mine.
I do wonder if after the Ether "flash crash" recently (https://blog.gdax.com/eth-usd-trading-update-5d8142b5bdc1), if Coinbase / GDAX instituted some sort of "circuit breakers" on their exchange to prevent this type of thing from happening again. Bitcoin and the other cryptos, trades at a pretty low volume relative to their market cap, so if a lot of people we're trying to dump large orders this morning to take a profit at the new ATH, I could see that causing similar issues that they may have been trying to prevent.
Circuit breakers do not matter if other exchanges continue to trade the same currency. It would just cause the exchanges that do the circuit breaker to possibly hurt their customers.
The only effective circuit breaker is some form of synchronized method, but will the different exchanges cooperate? They really should put in circuit breakers this week across all exchanges so that when really bad stuff the panic doesn't destroy tons of people. It will be bad, but at least if the trades can complete and things unwind in a moderately slow fashion, it won't just 100,000 attempted transactions per second with random successes.
Maybe not, I'm just speculating that GDAX or coinbase may have added this as some sort of "sanity-check" in their system (regardless of the insane volatility in crypto right now!)
In the situation with ETH, a bunch of automated margin calls happened because of one large order on the books, dumping the price down to 10 cents! GDAX/Coinbase repaid a lot of people who lost money then, even though they didnt have to. They may now have some dumb heuristics that slow down trades / whatever if a bunch of large orders are going through, or price delta over a short time period goes too high. I remember looking at the charts today on GDAX and the 5 minute candle went from 20k open to 16k close before trading halted! I also heard from others that there were a lot of ~19k sells of 100 coins each on the books and executing.
So yeah, just speculating that this may bave been the case of someones boss at GDAX/Coinbase telling engineering "please don't make us do that again", after they ate the cost last time! :-)
Haha, yeah I can't see the BTC crowd all being on-board with this.
That said, while I don't think the exchanges need to (or realistically could) coordinate on circuit breakers, I do think it's entirely reasonable for there to explore this type of mechanism on a per-exchange basis in the case of GDAX and other exchanges where margin is allowed, to prevent the whole "cascading margin call" thing that happened with ETH to happen again. That could take a lot of forms, including limiting the size of market orders to not eat 30% of the book, or not allowing 1 market order to trigger margin calls if (insert criteria here)
(That said, they could also just say "that's how margin and trading goes on low-volume, highly volatile assets" and technically be correct, but much like their decision to reimburse some traders, this is more of a business decision and wanting to keep your customers. Not sure what the "right" answer is.)
This is what I thought as well but haven't been able to articulate yet! It was actually on the opposite side -- a flash rise. I watched this happen and the order book on the sell side was getting extremely thin. The price rose thousands in minutes. I managed an 18k limit sell and bought back in at 16.5k
If this phase of Bitcoin is likened to each goldrush around the world, isn't it fairly inevitable that there will be hiccups? Robberies, banged together shopfronts, people using any tools they can find, etc. And the internet has long been a bit of a Wild West already.
Assuming they're adding thousands and thousands of customers quickly, I'm not surprised that their systems and support are lagging. Comes with the (Wild West) territory.
I wouldn't want to be in a business in which the worlds absolute finest hackers and crackers - the elite of the best of the best - make it their full time job - forever without end - to get into your systems and steal everything of value.
Banks have the advantage that if they see a huge spike in their outbound inter-bank transfers they can suspend traffic while they investigate it, and then if necessary call up the recieving bank(s) and issue a freeze on those funds. There's already established relationships and agreements for these things.
With a cryptocurrency there's nobody to call, you have to convince 50% + 1 of the mining pool to undo any major breach - which is almost certainly never going to happen.
I'm curious what the implications are that all these exchanges go offline whenever there is a huge spike in bitcoin.
That being said, anyone care to elaborate how the price of bitcoin is determined? I mean it's easy when there is a large central bank monitoring this figure. However, how exactly does it work for bitcoin? What makes it worth the $15K value when there is no "goods" to back it up. Are there anyone here buying bitcoin as an investment?
As others said, its determined by what people will pay for it. If you go to an exchange like GDAX (or any stock exchange), you can look at what is called the "order book". If you are a trader, you can act as either a:
- "Market Maker" - By placing an entry in the order book, which says something like: "I have up to 100 Bitcoin I'm willing to sell at $15,000.01 a piece", or "I'm willing to buy up to 100 bitcoin at 14,999.99 a piece".
- "Market Taker" - You can do this by looking at the book and saying: "I will take that deal". You can also do a "market order", saying: "I want to buy 100 BTC at market price", in which case, the exchange will match you with as many sellers as it takes to fulfill your order of 100 BTC, working from the lowest offered price on up.
The price is determined by either whatever the last trade executed at, or a weighted average of the buy side and sell side of the order book. If you hear someone talking about the spread, that is the difference between the highest buy price offered on the book, and the lowest sell price offered on the book. (2 cents in my above fictitious example).
The exchanges make their money by running the order book, matching makers and takers, and charging a small fee for the transaction. (On GDAX for example, market makers get charged no fee, market takers get charged some fraction of a %). The exchanges make money on the volume of trades going through them.
It can get a bit more complicated, but I hope that gives you an idea!
Is there a term for those who take advantage of the spread to make their money, essentially placing buy orders and then turning around and placing sell orders? Seems like there must be people making money off those who only "Take" deals and never "Make" them.
Those are just called "market makers". As a market maker you might be exposed to the risk of always ending up on the wrong side of the market over a longer time. For example if BTC is generally moving up, market makers might eventually have sold all their BTC and end up with USD. Sophisticated market makers will thus try to maintain a neutral position to avoid that risk. Some exchanges will even offer market makers a fee rebate to encourage parties to become market makers.
Exactly. GDAX and other exchanges charge less or no fees to the market makers to increase liquidity in the market. (Basically lessen the spread and increase volume on the order book).
From the exhanges POV this is useful because more orders = more money for then.
From the market traders POV (the taker) this is useful because a lower spread and more volume means there is less chance of buying something they can't later sell.
For the market maker, they make their money on volatility (aka the movement if the price). If they think the price is going up, they will offer to buy at a price higher than the current highest buy offer. This reduces the spread and increases the odds of someone taking their offer. They will then later sell those coins, and make a small profit on the movement.
If they think the price is going down, they will place a sell order cheaper than the lowest sell, and then buy back later, making small profit whole keeping the same amount of coin / stock / whatever.
Basically a "market-making" trader is less in it for long-term, and more interested in making money on the movement.
Aside: You may have also heard of "short selling". This is when a market maker is betting the price is going to go down, but doesn't have anything to sell. In a short sale, they will borrow the stock/coin from someone else (such as the exhange or their broker), and sell it, promising to buy it back later to repay the loan of said stock / coin. Easy way to bet on perfornance of an asset you don't owb, but dangerous since if you are wrong and price goes up, the person who loaned it to you will want you to repay it back sooner rather than later, costing you money!
What makes it worth 15k is that people will pay that much for it. The price is determined by buyers and sellers, trading with each other on markets called exchanges.
I get the "market" determines the price but I guess I'm confused with who is agreeing to the value? Is there a small amount of big players that are playing with the price? Or is there a general consensus that the price is $15K. And wouldn't there have to be people that we willing to pay this price? What prevents the exchanges from colluding and raising the price artificially?
Overly simplified explanation: You have to have a buyer willing to purchase at that price. Lets say the exchanges set the price at 100k when 3 minutes ago it was 15k. No one would buy.
A lot of the market movement is caused by people setting fixed prices on their transactions. Eg: Sell 5 BTC at X Or buy 5 BTC at Y. If X or Y is never reached the transaction never takes place.
Generally the exchanges just connect a buyer to a seller. You are not buying the exchanges coins but rather the coins of another individual/trader. The exchanges are just middle men facilitating the sale.
The exchanges make money on a per transaction basis on fees. If the market grinds to a halt because there are no transactions the exchanges make no money.
Now there are many possible reasons for the recent spike, some malignant such as potential pump n dump scheme or artificial inflation due to the upcoming futures exchanges and the desire to profit from shorting and some benign such as all the media attention and fear of missing out or just genuine belief in the potential of BTC as a currency or asset.
No idea which reason is accurate, could be all of them or none.
Hey, I hope you saw my above price about how the market works.
So to give you a little more info:
1) Yes, there are big players (aka "whales"), and the rest of the "small fish", all playing in the same pond. The interesting thing about cryptos is that the are more "money-like" than "stock-like" when it comes to trading. Meaning just like money, where I can buy a fraction of a $, I can also buy a fraction of a bitcoin. So there are a lot of smaller players who will buy, say .01 BTC for $150 right now.
But while there is a general consensus that bitcoin is $15k by virtue of people paying that much for it, the "whales" (people with a lot of money or coins), can cause issues because they can put a lot of volume on or take a lot of volume off the order book at once. This happened to Ether this year, dumping the price for a few minutes from >$300 down to about 10 cents for a few minutes! https://blog.gdax.com/eth-usd-trading-update-5d8142b5bdc1
In fact, there is some speculation that a lot of the rise right now is from Wall St. / big firms buying up a lot of coin, as futures trading on Bitcoin is coming in the coming weeks! (But I don't know enough to comment on that theory)
As far as preventing exchanges themselves from colluding... it would be difficult but not impossible, because again the price on the exchanges comes from the traders, not the exchanges themselves. (Theoretically it is possible that an exchange could do a bunch of "wash trading" / make up a bunch of fake trade volume to make demand look higher than it was, but given that there are a number of exchanges in a number of different countries, coordination there might be difficult)
>Is there a small amount of big players that are playing with the price?
We don't really know.
>Or is there a general consensus that the price is $15K. And wouldn't there have to be people that we willing to pay this price? What prevents the exchanges from colluding and raising the price artificially?
Well the exchanges could be running buy-bots but I think the more likely collusion is going to be big-players selling to themselves, if a big player is filling up both sides of the order book they can make it look like there is a lot of activity and drive the price up that way. Now is this that or is it organic? I'm not sure we have a good way to tell.
Naïve question here: since wallet addresses are fixed, couldn't you detect transaction activity unusually repetitive between a few wallets? Couldn't some clever learning algo detect anomalies such as what you hypothesized could be going on?
A Bitcoin address is just the hash of a public key, so generating new addresses is extremely cheap (on the order of microseconds) and doesn't require interacting with the blockchain at all.
People place orders saying they will buy it at x price and these are matched with sellers. I recommend checking out poloniex to understand how a crypto exchange works. Coinbase isn't actually an exchange rather they have to buy Bitcoin from elsewhere and then sell it directly to you and that's part of why the price on coinbse is always higher than on real exchanges
Look at it this way: these services are like banks, but their traffic is more like a news site. It's literally the most challenging aspects of both worlds.
simplistically, the exchange manages an "order book", which records the amount each person wants to buy or sell, the price at which they are willing to trade, and the time the order is received. if there is enough "buy" volume at a given price to fill a given "sell" volume, then a trade happens. the price of the latest trade is the "market" price. this happens for every order that has a volume and price matched "other side" of the trade.
there are many orders that are above or below the current market price. those wont get cleared until the trades that are closer to market clear, and they may not get cleared at all if the market moves the other direction
Generally speaking. For example, Nicehash, which had been chugging along for a long time, went belly up yesterday after what seemed like a regular maintenance period.
how many things can go wrong when the service is overloaded... looks like Coinbase doesn't know what stress testing or monitoring is.
* ID verification by either webcam or id pic upload fails, and eventually one reaches the daily quota, locking the feature for 24 hours (seems fair though)
* removing and re-adding bank accounts fails, the site says the account is already linked and it shows some ghost duplicates (i.e. the same bank account multiple times with different statuses)
* bank account verification fails, the site says the small amounts deposited don't match
* some buying operations fails due to API timeouts when collecting money from linked bank accounts
How can anyone reliably trade Bitcoin? With Coinbase immensely unreliable, I can't see how anyone would tolerate running algos on their exchange, given they could be in a bad position unexpectedly.
What are the other exchanges that you can trade BTC and are they more reliable than Coinbase?
Serious question, would it ever be possible to implement circuit breakers like the stock market does for Bitcoin? Without safeguards, and the nature of 24/7/365 trading of Bitcoin, this all feels like teetering on the brink of catastrophe.
I never want to reuse my comments from other thread but this time I just can't resist. No one really anticipated what we all are currently experiencing and Coinbase is not different.
Thanks, this is a really interesting way of looking at it. The change from $0-1000 is still not comparable to the other 2x gains. I looked up $500-$1000 and it looks like it took about 1000 days.
That being said, does price matter or does volume matter, in terms of Coinbase? Their infrastructure should be indifferent to price. I understand there's a correlation there, but daily trading volume would be the ultimate determinant of Coinbase's performance.
While I agree with you that the amount of users probably raised a lot to the point were servers are stressed out, there's something slightly off with the data you present: the latest entries are actually smaller percentage moves.
Here is the same data with (approximated) percentage moves:
$0000 - $1000: 1789 days
+100% ($2000): 1271 days
+50% ($3000): 23 days
+33% ($4000): 62 days
+25% ($5000): 61 days
+20% ($6000): 8 days
+16% ($7000): 13 days
+14% ($8000): 14 days
+12.5% ($9000): 9 days
+11% ($10000): 2 days
+10% ($11000): 1 day
+9% ($12000): 6 days
+8% ($13000): 17 hours
+7.5% ($14000): 4 hours
+7% ($15000): 10 hours
+6.5% ($16000): 5 hours
+6.25% ($17000): 2 hours
+5.8% ($18000): 10 minutes
+5.5% ($19000): 3 minutes
Anyway, it's more the volume that is of importance here than the price when it happens.
You don't need money "coming into" to increase value of Bitcoin.
All you need is back and worth trades in th market. It can even be market manipulation where some buy and sell between themselves.
Bitcoin has massive market liquidity problem. Once people want to sell, the price drops like a stone. This is why it would be convent to have "technical problems" or do DDoS attack to prevent sells and allow markets to calm.
Had to be a lot. Should have screen recorded it, but instead recorded with my phone. The order book entries and chart were going nuts. Entire sections of the book disappearing at once. The chart would go almost entirely flat several times.
Almost always you see minor movement on the order book except maybe sometimes at the actual mid point it may update a bit. The chart is almost always balanced until the past few weeks it's be HEAVILY weighed towards the sell side.
71K+ BTC had been already traded by the time it hit high 19k.
Technically, not much. Let's say people stopped trading except for two people that made a series of trades of .0000001 btc at $100k, then the price would be $100k.
Not much. 1% of asset allocation for an institutional investor is enough for that.
EDIT: You currently need $70m to bump the market above 20k on GDAX. That's a lot for retail investors but not much if institutional investors get involved.
You look at the "Depth Chart" -- it shows you how much people are selling (or buying if you go to the right) in a way that escalates as you get further away from the mid-market price that is currently executing.
right now, as i'm looking on gdax, the price is 16.3k/btc, and it would take 42m to fulfill all of the sell orders to push it up to 19k. that's not a lot (but it's only one exchange). i think <100m to swing the market by that much across all of the exchanges.
Tangent: it would be cool to have a browser extension that could notice that this data could be presented visually and offer to show you e.g. the scatter plot.
The insane rise in the valuation of Bitcoin. How many times we have seen a currency or an asset that represents this kind of momentum? Clearly no one has seen this coming (that soon).
They are rarely completely offline. But, they've been experiencing all sorts of "minor outages" and partial service degradation. And when you can't trade or transfer between wallets, the fact that their portal is "up" doesn't matter much.
Strange, the price data from coinbase never stopped working on tradingview.com. I wonder what was affecting the price changes when the order book was down?
The books are on https://www.gdax.com/, Coinbase's exchange intended for big trades. Coinbase is more of a consumer retail site. I assume the latter is what is going down, but it's hard to tell as they both seem fine at the moment.
As of now it is back up. If you take a look at the GDAX or Coinbase charts over the past 1-2 hours on a short timescale though (i.e. 1 || 5 minute candles on GDAX, or Past hour on coinbase app), you'll notice after the big dump, there was a large flat period where no trades were occuring, even though the site and app appeared to be up for a lot of people.
These past couple of days, whenever a coin soars a ridiculous amount, on any given exchange, that exchange seems to mysteriously go down. For example: Binance and Bitfinex were down for "maintenance" (or just plain down), while IOTA was experiencing a surge in price (from ~1.10 to 6$). Similarly this happens with Bitcoin today (it seems Quadriga had some issues and Coinbase; not sure about others).
Something is up, and it's weird.
My conspiracy theory: big money institutions are testing pump and dumps for when short contracts come into play. On the one hand, the cryptocurrency world wants to remain as unregulated as possible; on the other, a lot of individuals are going to get reamed by big whales.
Yea. I work as a software engineer at a company that sees huge swings in traffic on our product (by nature of the product). It's not uncommon for us to see 20x swings in traffic day-to-day. It's taken us a long time to be able to handle that and even then shit still goes wrong.
I'm much more inclined to believe the downtime is caused by scaling issues rather than some big conspiracy,
EDIT: Just to give some insight, one of the problems we run into is that usually only one issue is exposed during an outage. For example, lets say we have 10 functions (or services or whatever) in a row that all call each other. All the odd numbered ones have issues where they fail at scale. During the first downtime, function one will fail and start to throw errors, but this means the full traffic load isn't reaching the rest of the functions. By the time the problem is solved, traffic has usually died down, so function 1 is fixed, but 3,5,7,9 still have issues that haven't been exposed.
Additionally, at a quickly growing company like Coinbase, the code isn't going to stay the same for very long, so even once those functions are fixed, new issues are introduced.
It's easy to say they should be load testing at 10x their expected load to prevent these type of issues, but it's really hard to replicate production traffic. Users do weird and unexpected shit that can cause problems.
There's an article on the frontpage of HN from NYTimes that mentions Coinbase has struggled with the massive spike in traffic recently.
It's funny how quickly people jump to overly-complex or borderline conspiratorial explanations when the obvious is staring us in the face...
Coinbase might be a large well funded organization but it's still run by developers with giant backlogs of tickets and conflicting priorities to build new stuff vs maintain old stuff.
I have entertained using cryptocurrency for game betting at some point, while running some typical web stack. When I started looking into security and scaling, I realized there was a expensive and complicated overhead. When security is of very high concern, traditional performance seems to really suffer.
They are very well funded, and they can afford scalability. It's not that hard, given the number of transactions even at peak so far.
So yeah, I find these weird circuit breaker patterns suspicious. The worst part is "partial" degradation, when you can't execute trades, but someone else can. It's very easy to hide all sorts of irregularities behind these partial outages.
Not really. While most exchanges went down during the Nov 29th rally, most others didn't during today's, nor did they during the four June rallies where Coinbase also went down.
I think you can take off your tinfoil hat. Lots of price movement motivates more traffic to the exchanges, resulting in DDoS.
As the bitcoin market grows GDAX, Bitfinex, etc are probably going to have to borrow some ideas/talent from NASDAQ & other big electronic exchanges to stabilize things.
> Shouldn't an exchange be able to handle volume? Seems like a basic requirement.
Exchanges have incentive to handle as much volume as possible to make as much via fees as possible, but no site can handle infinite volume. The coinbase app has been climbing the iTunes charts into the top 20 the last couple days. Just before the site went down the price was running up by hundreds of dollars per minute, attracting tons of trader traffic trying to profit from it. Even if they have basic scaling measures in place for the basic infrastructure, they most likely experienced unprecedented exponentially increasing volume that was inevitably bound to break something
In my capacity as an infrastructure/scaling consultant I've spoken to multiple exchanges. Without disclosing anything I'm prohibited from, my strong expectation is that almost every exchange can't handle traffic spikes.
They're not all shitty me-too Rails apps built by junior engineers who figured "how hard can it be?". But most of the ones I've seen are.
> Shouldn't an exchange be able to handle volume? Seems like a basic requirement.
Exchanges are just websites like any other. They're build to handle a certain amount of traffic based on the average amount plus a little extra to handle spikes.
If they only get (And I'm pulling numbers out of my ass here just to make a point) an average of 10 trades per second with occasional spikes to 100 trades per second, how much infrastructure should they build? What capacity should they build to? 1,000 trades per second? 10,000?
There's a cost analysis involved. I think it's a bit unrealistic for them to have enough infrastructure to handle spikes an order of magnitude higher than their typical peak loads.
I agree, they _should_ be able to, but lots of times performance is a 2nd priority behind getting shit out the door - for better or or for worse.
I'm sure as they grow they'll harden their systems and/or do a total rewrite. In fact I wouldn't be surprised if they're doing it already since the big boys are starting to show interest with futures.
But what levels of volume? 100 quadrillion trades per nanosecond? Sure they could scale their infrastructure to those levels, but why? You overprovision, get estimate your worst-case scenario and work towards that. But in the end, if there is a huge rush, like a 10000% increase or more, even the largest exchanges would break down.
US stock exchanges don't run websites on which you can trade. However, during the flash crash in 2015 I was unable to place trades on either of the two brokerage sites I used at the time (USAA and Vanguard), probably due to other people rushing in to trade like myself.
I imagine they've spent a lot of time already beefing up their infrastructure, you can anticipate high loads but until you know what that looks/feels like, you cant compensate for it. They'll get there eventually. It'll just take some time.
These services are like banks, but the traffic they handle is more like a news site, due to the extreme volatility of cryptocurrencies. Super-security combined with extremely large traffic fluctuations.
Whoever is on PagerDuty at Coinbase these days - I feel for ya.
> big money institutions are testing pump and dumps for when short contracts come into play
Big-money institutions won't do this. Random individuals taking advantage of the fact that lots of value rides on relatively little volume? Much more likely. Combined with the fact that the exchanges, intentionally or unintentionally, lock liquidity when volatility spikes and it's 1907 all over again.
Given the incentives involved (and history of exchanges, everywhere, before regulation) a good fraction of that is wash trading. (A better measure for the risk at hand is the effect of small out-of-market bids and offers on the overall price, i.e. how much it costs to move the price $1.)
Can you explain? You could also argue that when futures are listed on the 10th, there will finally be an easy way to go short on Bitcoin. For instance when you want to hedge a long position, or simply because you want to speculate on Bitcoin going down.
So the listing of Bitcoin futures on the Chicago exchange might not be a good thing for the Bitcoin price at all.
No, No, No! Conspiracy theories attract attention because they make the world seem more legible, when it's not. In this case, exchanges are not being attacked by shadowy corporations for short contracts.
Attention from price surges increases visitor traffic, and exchanges are understaffed or unprepared for the 10-100x increase in traffic volume. There's no need for complicated stories, evil corporations, or any of that nonsense. Stick with straightforward cause and effect.
No, No, No! Don't pay attention to that thing! Look over there!
Seriously though, it's worth considering that this kind of thing could be possible even if we don't know whether or not it has anything to do with these specific outages.
You are coming off as very emotional/desperate in your attempt to convince people to act cool and logical. It's not very persuasive, and if I could downvote you I would.
I am not excusing their behavior as they need to be able to handle it, especially when their server status affects the bank accounts of real people. But I feel that it is understandable with out condoning it. The simplest explanation is likely correct in this case.
I don't think many of us could have predicted the volume and interest in BTC a year ago and neither did Coinbase's purchasing and infrastructure teams.