As a fellow software engineer, I feel for the people who work there.
As a customer who lost a decent chunk of money yesterday (and today) because their system was down, I'm angry.
I hate to say this, but I do not think they're competent enough to continue to operate as a broker. I, for one, will move all my money off as soon as I can. I'm also going to file a complaint with finra[1].
Right there with you. I only use RH for options -- so thankfully not a lot of money in there for me, but I'll be switching everything over to tastyworks.
It has a nice GUI for different option strategies. Great website and mobile. Fairly cheap commissions too. If you are just buying/holding shares probably not the right platform.
Because it makes the assumption that they are still well intentioned and want their product to succeed. Even well paid engineers feel stress. Additionally, overtime pay seems unlikely :)
"Why should I pay overtime for the system you broke?" - The manager who ignored their engineers requests prioritizing higher quality testing and more redundancy
I've been defending Robinhood since the beginning but this is just too much. I will start looking at new brokers this afternoon.
At this point for me its not so much about the lost opportunity as it is about what the hell actually exploded so hard that they need to be down for 2 consecutive days. From my perspective, it's looking less and less like some one-off technical or hardware screwup and more like a fundamental limitation of their core architecture.
I actually switched most of my "real" investing to https://www.m1finance.com/ a few months ago. It is also commission-free, but unlike Robinhood you can do IRAs as well as taxable accounts, and it automatically reinvests dividends. One interesting difference is that they limit buys/sells once daily (twice if you pay a small monthly fee). Day traders won't like this, but for buy-and-hold investor not trying to time the market to the minute it makes sense to me. As a developer I'd think it has to be easier to code than real-time. I feel like it's essentially a cheaper Betterment/Wealthfront. I keep a taxable "short term savings" account in 90%bonds/10% stocks, and then my IRAs in 10%bonds/90%stocks.
I haven't looked into the technical credibility of this at all [EDIT: I should have more strongly indicated my doubt here], so this isn't any sort of condemnation on my end, but I thought the Leap Day theory was interesting:
It's starting to seem like they use some sort of message bus to tie everything together. If the servers actually processing messages can't handle the volume, the entire show comes grinding to a halt. I've seen similar things in other industries where the entire system looks like its perfectly healthy, and then before you know it 100% of your systems are down because you simply can't push peak message volume or because one participant is infinite-looping messages.
Their director of infra tried to recruit me and basically said that they wanted to replace every relational data store with message queues instead... that struck me as a bit weird and overzealous.
I am seeing this parroted everywhere this comes up and it is very unlikely to be the case. For one we have no idea what that API call is doing. The date parameter could be an upper bound, ie. it's just asking for all of the most recent data. In addition, Robinhood was up over the weekend, after the leap year occurred, and was even available early on Monday before the market opened.
Robinhood only trades in the US and the US markets are closed on Saturday and Sunday which means there were no live orders on those days which in turn means that certain code paths were not executed and errors were not thrown on Saturday and Sunday the two days when it would have been detected.
There was a discussion last night about this. Some of the more technically competent posters dismissed it because it's unbelievable that a financial platform would roll their own date-time implementation.
I work in finance, and at a previous employer (not Robinhood), I partially rolled a datetime implementation. Mostly it was a wrapper around Boost Date Time [0]. It was a facade that smoothed out the interface and implemented some missing functionally, like a cross-platform strptime and loading of the Olson timezone database.
I spent a better part of a year working on it (along with other things). Modeled the interface after Python's datetime module (which I think is one of the simpler and easy to use date-time libraries across various languages I've used). More than $20B USD trades using that library.
The motivation was the firm used to use RogueWave's date-time facilities, but we moved away after they jacked up the licensing. Think we used to have a site-wide license, but they were moving to a per-core licensing, wanting something like $2K per core annually.
Needless to say, testing was extensive. Overflows were found in Boost in far distant dates, had to work around those. Tested against a ton of historical dates. Tested against Northern and Southern Hemisphere daylight saving time (a lot of people don't realize that Southern is inverted from Northern). Learned a lot about timezones and the history of timezones along the way.
While I agree in spirit (Occam tells me it was just unprecedented load), that explanation doesn't make any sense to me in terms of disproving the Leap Year theory. The mistake wouldn't even have to be in their code.
As for believing/unbelieving, I mean, I've seen dumber mistakes in financial products I've worked on. Not as high stakes or damaging, but definitely dumber.
That means people should reevaluate the weight of the opinions of those technically competent posters - we have a screen shot that robinhood did roll their own implementation because yesterday was March 2nd and its app was requesting March 3rd.
That at least means that some portion of their stack used roll your own datetime library. That would not actually be a problem as long as the entire stack got the same library or the same rules for datetime. The problem is of course that it probably did not so the libraries were not bug compatible which of course caused errors and those errors need to be handled, preferably every fast.
Error load and error handling is the least tested part of every system because it can only be properly tested in production and, unless your company embraces the Chaos Monkey approach to testing, the C-level would have a heart attack when anyone proposes doing it.
The issue is going to end up being related to some sort of date being injected by the front-end and propagating that incorrect date into the infrastructure.
Somewhere within the infrastructure there's going to be an assertion such as
if (max_drift_delta > delta(order_live_date,system_live_date) {
# oh crap, something is completely broken in our system where did this come from?
blow_up("terrible things are happening! how did we get this order")
}
which is an excellent and correct catcher for "terrible things are happening" since those things should never happen. That blow_up() code path is likely to be very expensive which kills performance of the system, which in turn means that it no longer can handle the load.
And since RH has lots of people who use apps, it is not that they can just push an immediate bugfix.
Something similar occurs to crypto exchanges. They go down during days with extremely high volatility. During Q4 2017 most of the crypto exchanges were fighting hard to stay online, sometimes in precarious conditions, sometimes with execution delays of up to a few minutes.
I'm a QA engineer and this is freaking infuriating. I just went on Linkedin and 0 QA engineers. Just some QA associates that have no tech backgrounds. And one QA Engineer opening. I wonder how long it's been open? How does a trading platform not have a big team of QA engineers? Some dev managers, devs are so freaking arrogant. They think since they aren't coding, they aren't smart and they aren't important.
This isn't the first time that it's happened. Robhinhood should have been prepared. This is unforgiveable.
As someone who also needs to explain the importance and nuance of QA engineering to my startup partners, do you have any go-to resources you use to describe/sell the position? Genuinely want to expand my ability to speak to the role.
Point them to situations like this. Maybe small e-commerce sites can get away with it but financial and health tech companies should always double, triple, check everything. Move fast and break things is fine for new startups but once you have a significant amount of users, the reliability of the product should be the main source of pride for the company.
I know not every company needs it. I was the second QA engineer at a food-tech startup in Boston 5 years back. QA got laid off after 10 months but they are still successful without QA right now. They hire great devs and have a great process. At the end of the day, if the site stops working, people will just get some late food deliveries. One the other hand, I've a friend working at Ford in Michigan and he says they have no code. You have hundreds of engineers and everyone manually tests their own code. He says sometimes he will test for a couple of minutes and just push his code. I don't blame him. Devs are great at writing code. Let them write code but have someone else test their code please.
I've a small options position open on Robhinhood. I'm not too worried about it but I imagine some people have tens and hundreds of thousands. I've read some rumors that this outage was caused by the leap year timing and it happened 4 years ago too. This is just unacceptable.
Genuinely curious, are there a set of a disciplines a software QA engineers has to have? What tools do they use? (I only know they do software testing, but this could mean a lot of different things)
I understand what Devops (a similarly amorphous term) people do, but I've never had a clear picture of what the day-to-day life of a QA engineer would look like.
What's wrong with people who think that they can have SDLC without a proper QA? I've seen this many times in the SV. Developers thinks that QA is just waste of time and their stellar code doesn't need any testing, because: "Hey, if something goes wrong I can always roll back". I don't buy it.
Most big SV web companies employ SRE, not QA to manage and improve the reliability of distributed web systems. Robinhood currently has SRE positions open so it’s not as if they aren’t thinking about this at all
Often it's not dev managers, but a level above who don't prioritize QA engineering because they see it as a cost center and find it difficult to quantify the risk. If it's not leading to growth it is de-prioritized until fires are burning.
A lot of the trouble is that good QA -- and good engineering practices in general -- usually comes down to some equation like velocity x quality = $. If you want to increase your code quality, you either have to slow your velocity or jack up your $'s -- and usually you end up doing some combination of both, going a little bit slower and spending a little bit more money.
And if there's one thing people hate more than going slow it's spending more money.
My reformulation is mostly out of ignorance, but since this is the internet, I will contend in a fit of pique that in modern software development, velocity (which is basically scope / time) is more interesting than the two factors individually, because so many of us live in a nightmare world where there are not finished projects, just continuous iterative release cycles.
Regarding getting some money back from Robinhood, here's an interesting comment I found at Reddit:
> To those of you who deposited funds to Robinhood and are now down your deposit or more, these are technically ACH transfers, and if you genuinely believe Robinhood defrauded you, or otherwise failed to provide service as understood, you can issue an ACH chargeback through your bank’s web portal or by phone for any funds deposited in the last 60 days.
> RH may try to call you into arbitration, but more likely they’ll just ban you from the platform. It’s not a terrible idea if you plan on leaving the platform anyway. Just make sure you can substantiate your losses or they might have a fraud case.
> [If RH dings your credit], you‘re able to dispute with the credit reporting agencies or, if the agencies won’t delete, hire a debt lawyer to resolve. I doubt they would even if able. Any representation they have is likely about to be drowning in FINRA and arbitration prep, some people keep obscene amounts of money on their platform. They also likely can’t furnish any signed documentation of credit pending payment unless you signed up for margin trading.
> If the funds haven’t been deducted from your bank account or are in a pending state, you may also be able to just issue a stop payment. Again, anyone who tries this should ensure that they can substantiate losses related to the payment they stop or chargeback.
It's a shame. I was rooting for Robinhood, since they really did disrupt the brokerage industry. I suspect two days in a row of this will be a death knell for them.
I don’t actively trade stocks but I’ve looked at it. What exactly does robinhood do that etrade didn’t? (Other than the shiny app and front running being a formal part of the business model.)
As I mentioned, they were the driving force behind the industry moving to zero-commissions. Also, I do really like their app compared to Fidelity, Schwab, Vanguard, and e-trade. But I'm probably a Fidelity customer for life at this point.
For a while they were the only no commission player. Now that virtually everyone offers this there is not much to differentiate them but having a simpler interface. On the other hand the interface is simpler because it offers fewer securities/derivatives
Since it's Day 2, def appears to be an architectural scalability issue, not some bugs. Quick scan of their engineering blog shows the following stack:
-Zookeeper, Kafka, Spark, Airflow, Faust (Kafka Streams in Python)
-Kubernetes (was/is SALT + Terraform)
-AWS Aurora, PG, ES, Influx
-Presto via AWS Athena, Redshift
-homegrown solutions for a data lake & managing k8s microservices
Fwiw, their team does appear to have a rigorous internal post-mortem process (SEV reviews) [0] drawn from industry best practices.
Let's hope a public one will be made when issues are resolved. (FINRA will be asking for one anyway.)
Not piling on. I genuinely feel for all the infra engineers staying up all day/night fixing this.
Is this a technical issue with the platform or underlying financial issue with their company and margin system?
I would not be surprised if a “move fast and don’t test things” approach to margin stress testing would collapse in the market we’ve had for the past week.
> The outage was not caused by a failure to code for the leap year. We had instability in a part of our infrastructure that allows our systems to communicate with each other.
I actually don't think any sort of gatekeeping to do whatever the hell you want to do with your money is the broker's business. Sure, don't allow naked calls and other margin shenanigans, but for the most part - these are fully functional adults - if they decide to gamble with their money, so be it. Let them do it.
I tend to agree with allowing people to do whatever they want with their own money, but I just cant picture a broker like IB for example allowing this sort of thing without much more serious vetting of the client.
You probably don't know much about IB then. There's no vetting at all, by IB or any other brokerage, except making sure that the money is in their account. They have margin calls for a reason.
Please move to a real brokerage if you use Robinhood!
> Why?
EDIT: Because their systems are garbage and their support non-existent. You can do better literally anywhere else (Fidelity, Schwab, Interactive Brokers).
Why ? Somewhat serious question - what will a real brokerage offer me on my tiny (< 1000) investment - we're just testing waters (at a bad time no less)
Blanket statement like that without reason would equal please switch to self hosting git instead of using github.com cause it went down last week or so - both have their purpose / skill level / cost associated with them. So a 3rd party app went down for a bit. Happens often (as most of us know)
"Investing" and "Stock Trading" are incompatible in the same sentence, unless combined with a phrase like "does not involve".
You can buy and hold, or "Invest", on any platform which, in my opinion, has to meet three minimum requirements: It should offer commission-free trading on a variety of low-cost index funds, it should offer a variety of investment account types (at minimum, taxed accounts, Traditional IRAs, and Roth IRAs), and it should be stable.
Robinhood only meets one of these three criteria. There are dozens of other brokerages out there which do meet all of these criteria: TDAmeritrade, Schwab, Vanguard, and Chase are four great ones.
Robinhood is on the same level as a betting app for sports; so, fine; use it if that's what you want. But do not even mention the word "Invest" in the same sentence as Robinhood; its grossly irresponsible.
Sure, I'm defending starting with less than $1,000, not doing so on Robinhood specifically. Your suggestions are good, I use Schwab and Vanguard myself.
I wouldn't advocate anyone trying to build wealth to pick individual stocks, but I do think if someone were investing $1,000 to get their feet wet I'd recommend they put a bit in individual stocks. The financial system is complex and I think it's worth stripping away one level of the complexity for educational purposes. (It's typically also a humbling experience.)
I agree, this is how I started. It was well worth a couple hundred dollars in losses 2 years ago to learn the fundamentals. I considered it the cost of learning that I've more than recovered.
Now I have a much larger account, and I'm trading / investing with a much lower risk profile, because of what I learned on my >$1000 account. I consider that the exact opposite of wasting time.
> Investing under $1000, especially with commission-free trades, is a good way to learn the basics mechanics of stock trading
Individual investors are at a huge disadvantage when it comes to intraday trading.
For everything other than a child's hobby account, intended to teach emotional stability through gains and losses, a <$1,000 stock-trading account is value destroying.
you want me to risk > $1000 when I don't know what I'm doing with a real brokerage that I don't know what their benefit is over something like Robinhood because.... ?
I'm willing to loose a couple hundred to learn and understand something vs giving someone I don't know thousands and "trusting" their opinion.
More so - testing the waters myself may not make me as good as someone who does this for a living - but maybe after a few hundred and a few months, at least I have a better understanding of what a real brokerage tells me to buy than to just blindly say take my money and quadruple it.
Brokers should never be telling individual investors what individual stocks to buy, at least not anyone with less than ~$500,000 in assets. If a brokerage is giving you buy/sell lines for individual securities, that's a red flag.
A good investment platform (or adviser) guides you in portfolio management. In encourages long-term strategic thinking over short-term trading highs. The former builds wealth. The latter lines professional traders' pockets.
Why would you play with real money to learn instead of doing simulated trades? You can set up a paper trading account with Interactive Brokers for free. If simulated trading doesn't feel "real enough" because there are no stakes, then I don't think it's about learning.
That education, for a child or an adult, is primarily what one gains from a <$1k trading account. That, alone, is worth more than $1k.
Also, one need not trade a small account intraday. I make small trades, but they are always with a multi-year perspective. Give me limit orders and small commissions, and I am happy.
(N.B. I use a different broker/don't have a first-hand perspective on Robinhood.)
Unrelated but usually you should place a market order usually, not limit. Limit orders take longer to fill, your idea of limits is probably wrong and will lose money compared to a fair market match, it's not worth taking the risk that your order won't be filled, etc.
If an order isn't filled at a price at which I find reasonable, that is okay with me. A market order will fill at any price. I learned that lesson the day that a market order of mine filled at a price I deemed unreasonable.
The key to avoiding faffing around with limit orders not filling when you want immediate execution is to place a reasonable limit that accounts for the day's volatility. At other times, I'll place a limit order and let it stand for weeks. When it fills, the counterparty and I are both happy.
Different strokes I guess. Getting into/out of the position is my top priority and I see more downside in failing to fill the order than in failing to shave a penny or two.
Totally agreed on different strokes. If speed matters most, there's nothing like a market order.
I'm rarely using limit orders to shave pennies if I want quick execution. I use them to prevent the unexpected. Under normal market conditions, if the limit is set 10% beyond the expected clearing price, it provides me with free protection against a completely unexpected surprise. Something will have to have gone very wrong with my investment strategies if I'm desperate to buy/sell at any price.
In the long-game case, if I purchase a stock at $0.95 that I think can sell for $0.99, I'll immediately place the limit-order for sale as soon as I've made the purchase. No need to hide my hand -- I'll be happy if you want to buy it at that price (and I might get faster execution by being earlier in the order queue).
If you are investing in ETF/indexes then RH being down is not really an issue... and if you are looking to trade at those dollar amounts, then its a great deal. I fall into this category and RH being down has affected me in no way.
If you are trading much more frequently/short term, or WSB style options, then yea, I think that a proper brokerage (IB) is likely better for you.
Also perusing twitter --- lots of people were planning on executing that _perfect trade_ if only RH was up... lol
I have no experience with it directly, but Robinhood is also widely regarded (in at least some circles) as having very poor execution. Meaning you're potentially not getting orders filled, or getting them filled at a sub-optimal price. They're also known for having "tech startup" level tech support if there _are_ issues, unlike more established brokers that allow you to talk to a real person almost instantly, allow you to trade over the phone if needed, etc.
They're also known for mispricing essentially _every_ morning, showing nonsensical pricing and account values for the first 30 minutes of trading. Overall, their software is clearly lagging behind something like Think or Swim from TDA.
Robinhood kind of made sense for low-value accounts that wanted to avoid commission. They've changed the game a bit in that most brokers offer low to no commission now. I'll echo the parent comment, I just don't see the point in using them given that most of the alternatives are better.
As someone else alluded to, so you don't have to move later as you grow. Moving isn't the end of the world (although from what I understand RH charges $75), but it's just one more thing that's annoying to do.
If you want to use one of the better tools, see TDA with ToS. Schwab has great service and is a great bank for a one stop shop kind of experience. IB is one of the least expensive ways to trade if you meet their minimums.
RH OTOH has shown time and again they play a bit too fast and loose for comfort when money is involved. They have had to pull their entry into banking a couple of times now. The app seems to have issues at the worst times. From what I have read, support is abysmal. Not someone I want to deal with when it comes to my money.
First off, we shouldn't settle for shitty software, ever.
Second, now your investment might be small, if it grows, and represents a portion of your savings you might not be as delighted as today, to see them being offline during such a busy day.
To a retail investor who just wants to buy some ETFs and shares of AAPL to get their feet wet, none of these things matter, so this doesn't really answer the question.
The problem is most brokerage firms don't have two days of downtime where you couldn't trade at all, while the market is in such huge fluctuation.
EDIT: Why am I getting downvoted for this..? That's the reality of the situation here.. If I missed something, please reply instead of just downvoting.
Some context (I had no clue what the site is about):
Robinhood, a pioneer of commission-free investing, gives you more ways to make your money work harder. Robinhood, a pioneer of commission-free investing, gives you more ways to make your money work harder. It's Time to Do Money.
Those are usually updated manually by a human potentially hours after the outage started. It would probably be better to just probe their backend APIs directly
Related: I noticed Vanguard's site was having trouble handling requests last week. It was generally slow and would frequently error out or log out. The site is running well now and it appears at least part of the solution was to switch back to a previous version of the UI. Maybe two years ago they switched to a more responsive, web 2.0-ish interface (which, to my eyes, significantly reduce the information density of their pages in favor of large text and looser spacing). This says to me that someone implemented a poorly performing UI on top of a well-performing backend. Would be interesting, though unlikely, to get a technical response from Vanguard on what happened.
My company's 401(k) provider's site was also experiencing significant issues. Seems like last week would have been a fun one to be a system administrator in the investment/brokerage sector.
Well, "free". They hold your money and your stocks, which gives them opportunities to make profits. They also play middleman (or rather, sell the rights to be a middleman), making a profit on the difference between the asking and selling prices.
Robinhood has the worst customer service and security. My account got hacked, I changed the email, phone number and enabled 2FA and it got hacked again. I was trying to resolve it for a month.
I really don't see why anybody would use Robinhood instead of an actual brokerage, now that all of them are free.
Thread from exactly 4 years ago, March 2 2016: https://old.reddit.com/r/RobinHood/comments/48mep4/robinhood...
Thread that pointed it out: https://old.reddit.com/r/wallstreetbets/comments/fcoaev/and_...