The OP details how poor software engineering practices brought down a 1.4B market marker with 1400 employees in 2012.
Some of the issues mentioned include:
- Keeping synthetic test data generation as part of a production build.
- Keeping dead code for years.
- Re-purposing a feature flag.
- Refactoring without regression tests.
- Manual deployments without peer reviews. They forgot to update one of their servers with the new code.
- Automated alerts sent via email were ignored.
- Rolled back to a version of the code running on the server they forgot to update, making things worse.
- Rushing out a release without proper software engineering hygiene.
The article suggests improvements that could have prevented the chain of events.
For those here who are in HFT circles, have things improved after the Knight Capital Group debacle?
I worked in algo trading for years, eventually got out because quite frankly the level of risk I was carrying on my shoulders everyday for what I was being paid were just way out of whack, I at least personally never got the huge pay days that people talked about until after I left finance for more pure tech. Interestingly, I worked at Knight and my team pioneered trying to blow up the firm, but that was in 2004, and things were much friendlier- instead of front page news, it was a small blurb on page 3 of the markets section of the WSJ.
Anyway, I still have friends in that business. It hasn't really changed, they have too few people covering systems that are quite complex and while there are checks and such, no one really understands things entirely from end to end in detail that can prevent all problems.
I will never invest directly in an investment bank- either through carelessness or maliciousness I could have easily caused a 9 figure loss, if not more, and there were probably a thousand other people in the same position.
When I read the detailed writeup around this a few years back, I think by far the biggest issue was reusing a tag that had been previously used to denote which strategy to use. I understand why they may have chosen to do so, at the Big Bank I was working at, getting a new fix tag to be passed through all the layers properly would involve at least two other teams and coordinating releases and probably several weeks worth of meetings. If you just reuse an old value you can avoid all that since everything is already set up.
I appreciate your comment about pay. Recruiters will often tell me "it's finance so of course the pay will be substantial." Then when we get to talking numbers they're like "300k a year". Oh, you mean the going rate at a FAANG? And I have to move to New York or Chicago, work more hours, and actively work for people who I know are taking home paychecks with 7+ zeroes on them? Come on. Sometimes it's 400 plus bonus or whatever, which is based on fund performance and yada yada. But it feels way off. I had heard so much about the staggering paydays at these places but it seems you need an ML PHD or some trading chops to be part of that.
Yeah, pay at the big banks is shit really, especially when you consider the utter lack of work/life balance. I left in 2013 making 150k, which was supposed to be supplemented by a ~40% bonus for the level I was at, but each year was "well its been a tough year..." and after getting a token amount one year, and then zeroes the next 2, after working 50-60 hour weeks, I was like I am not only done with this place, but this industry, and left for a 50% pay raise, my TC is now 4x where it was in those days. A neighbor of mine is more or less sitting in my exact seat there, and is somewhere in the 200-250k range.
That said, I went back to finance to work at one of the premier hedge funds out there, and they actually lived up to their expectations in terms of comp, that place was more like a tech firm though than any other firm I have ever worked at aside for maybe Knight. 8% annual increases were normal there. You can look in my post history back to 2018 if you want the name, I recently left after 5 years there and just want to stay out of their crosshairs- they monitor social media aggressively and there is deferred comp at stake.
At big banks, there are really only a very small number of people who are in tech that are getting paid- you have to know which questions to ask- where is the bonus pool coming from- are you "in the business" or the tech pool, which is a second class of citizen. I would have to be in a pretty bad place to ever consider going back to a bank, it was borderline abusive... always dangling the prospect of that big check that would make it worth it
I spent a few years at an investment bank, not in the US, and the only people on serious money were some of the top managers. But my overall opinion of these people were that they did very little but the lower downs I met were some of the most talented people I ever worked with.
The top ones would spend all their time traveling the world to the offices and meeting with staff in each location and the sending emails to the rest of the department about what the staff in that location were working on. They would harvest ideas from the staff as they went and then present that as their own or approve projects that staff have suggested to them.
I really didn't see how they were worth the $5M they were earning since they didn't come up with the ideas for what would be done and didn't do any real work.
>> Then when we get to talking numbers they're like "300k a year".
> Yeah, pay at the big banks is shit really
I would say that a $300,000/year salary is really quite far from "shit". Many jobs are stressful, demanding or unrewarding but rarely do they provide such a salary, and quite often they offer hourly wages hovering around minimum wage with poor benefits. It's worth taking a step back once in a while to think about this, because it sounds like some people are a little out of touch with how the majority of the world lives
I would agree, but I really don't think many people are making that kind of money at IBs. Things may have changed, but as I mentioned, a neighbor who is pretty much in my seat- well technically one level up as a director (though title inflation has just made pretty much all titles below MD worthless), and was not pulling 300k. The new guys on the desk were not clearing 100k, and even "senior" guys like me were making a wage that just barely put us in the ownership class, and we were sacrificing a good chunk of our lives for it.
And again- these are the cream of the crop, this is the team most people wanted to work. I was one of the first to leave, but all those guys are now making multiples elsewhere that I am aware of. The point is that there are just better opportunities out there where you get to actually enjoy your life. I hear you on other jobs being stressful and demanding, but I have not really worked at a job where I was wearing that stress 24/7- well typically nothing traded on Saturday night, so nothing would ruin that, but outside of that, I could expect a call at literally any time of day- the markets were trading 23x6.
This is a welcome bit of perspective, so thank you for that.
I often find myself complaining about things at work like competing priorities or wearing too many hats and it helps me to ground myself by thinking about past jobs at minimum wage or even watching what people making less than a fifth of my salary have to put up with on a daily basis.
I believe that technology and other workers who are paid these salaries are actually just the only workers lucky enough to be getting a fraction of what is their fair share. You shouldn’t let a CEO walk with 300x their workers wage and say “well at least I’m not in poverty like those other workers”. You should almost always fight for more.
Most quantitative hedge funds and prop trading firms are now following a very tech like culture since they realize now that technology is just as important as the strategies. To get the best engineers, especially from FAANG, they need to have a similar culture otherwise they'll have a hard time getting new hires.
Frankly, a lot of what people refer to as "prop trading firms" and "quant hedge funds" on here are just market-makers...they are taking very little to no risk, a lot are just riding the wave of ETF growth. Even the ones that are running alpha, I have heard of a few strategies, and they are largely what you would expect: low-edge, crowded trades (frankly, a lot of it is LTCM-style stuff).
That is why the business has become more tech-like, because actually taking risk is...quite risky. If you are going for alpha, you are trying to hire one guy out of thousands, that person knows what they are worth, etc. It is far easier to hire lots of devs on low wages and do the grunt work jobs that are less lucrative, but don't require being able to actually work out if someone is a decent earner.
Man Group has their own department at Oxford Uni, I think Winton hires people out of their department at Cambridge...Man Group's shaky record is legendary (they are doing everything now that blew them up in 2008), Winton was top tier...not anymore. AQR is another one, although a more traditionally finance quant approach...it all turned out to be pure beta. The hybrid approach of hiring devs or ML PHds to generate alpha has only ever worked accidently.
The firms that have been printing money from quant investing over the past five years have all been traditional hedge funds that incorporated quant methods into a fundamental process. And I expect that will continue because, bluntly, these firms know what the price of a security should be better than some busted factor model.
No. Because that isn't what they do (I am talking about fundamental investors: Marshall Wace, Point72, etc.). I have no real idea what they do or do not do but they trade over shorter time frames where there is no change in fundamental information (i.e. they are trading based on liquidity signals which is, essentially, what market-making is now).
Sounds like I had a similar experience to you, except many years prior. I actually worked on a desk where there was a front page-headline scandal, partly caused by lack of investment in tech. I would never recommend anyone work in finance based on my experiences, but most people get very excited and assume it was fun and lucrative work. It was neither.
I heard, on multiple occasions from senior leadership, that industry practice was to categorize all dev work as IT so they could justify lower compensation. Pay was, just as you pointed out, not the main problem with this setup. There was no respect for other people and individuals were frequently berated or humiliated for no reason.
One interesting observation: despite the fact that I am very well qualified to work in the sector, it's very rare for fintech recruiters to reach out to me. They must know that I'm far out of their pay grade now and will never go back.
The attitude that finance pays more is a leftover from a previous era. 10-15 years ago it was true: the profits from HFT were so also way, way bigger and split up amongst a much smaller group of firms.
Now those firms are all in a completely competitive industry squeezing each other for basis points.
Meanwhile the definition of a FAANG is that it has an effective monopoly, and these companies are taking in way more money than the HFT industry. (Netflix is losing its monopoly but we can’t really drop N from the acronym without a replacement..)
Tbf, most of us don't really prefer to be called as HFT's but as Market Makers. Different name, but we still use the same ultra low latency techniques to get the job done.
I worked in option AMM when the term HFT started getting thrown around in some articles, and was discussed as this uber secret hush hush thing, and I was interested. I was then astonished when I kept reading and found out that it was what I had been doing- it was just another name for market making really...
Yep. At the end it is market making, in markets where speed is of the essence to be competitive. Funny thing is, now firms in the industry want to be termed as Market Makers rather than HFT since the media has turned HFT into a very negative term.
Back when Flash Boys came out, we all started calling each other Flash Boys. There was then a serious proposal to rename one of the strategies (the hedger, for added irony) Grandma Annihilator 3000.
When the KCG thing came out, one of the new grads was renamed “Power Peg” for a few weeks.
I miss the 2010s. It was a lot more fun when people leaned into stuff instead of being scared all the time.
There is certainly much variance between the firms, especially the sign-on IME. People I know have turned down HRT core dev for big tech because their offers were unimpressive.
I think an interesting target for comparison with big tech is Jane Street, since their culture and WLB are good, so the main QoL drawbacks of finance don't apply. A new grad will get ~300k at Jane Street, though probably not with this large a sign-on.
This is interesting, didn't know this about HRT Core Dev where offers were below FAANG. My understanding is that core devs are basically the folks who work on all the low latency stuff, so they'd be pretty well.
Jane Street, from what I recall, is 300K (base + bonus) and 125K sign-on, and also it is non-negotiable. No idea what their numbers are like for experienced hires from competitors.
Honestly, that depends on the firm. There are same that do pay very well like this, eg: HRT, Jane Street (they are not an HFT though), Headlands, Radix. Some others like Jump, Optiver the pay varies depending on whether it's front office or back office.
Where I work at, the new grad offers are slightly better than FAANG, but the growth is very good based on performance, we also pay very well to people coming in from a competitor.
Why pay more than market rate of a replaceable ML person?
Is there any realistic path for a demonstrably smart and hardworking person into that 7+zeros club? Evidence suggests no: leetcode grinders and FAANGers are not in that club, and most of them will never even make it into the 6+zeros club. Net wealth -- sure, but not income.
It's all about making $$ for the firm. If the strategies developed are very profitable then 7-figures is definitely reachable for the researchers at a prop trading firm.
Yes, exactly, and nomenclature borrowed from the parent. (This is another throwaway, and I am responding.)
> > "...all about making $$ for the firm. If the strategies developed..."
The topic here is on offers to new hires -- not value delivered after time and work invested. Offers at market rate -- even in the FAANG world -- are not at 7 figures (6 zeros), definitely not at 10x that, and certainly not before the new employee has done anything worthwhile for the firm. There are individuals who do get such offers, but they are rare, they have done remarkable things previously, and they have a reputation they can bring to any conversation about compensation. Their market is not representative.
I am sorry, I misread it as getting to those numbers as an employee at a trading firm rather than as an applicant. I agree with you. The very, very rare cases where this has happened is for people with exceptional records elsewhere, and many times they are joining new places as partners rather than a regular employee.
> I understand why they may have chosen to do so, at the Big Bank I was working at, getting a new fix tag to be passed through all the layers properly would involve at least two other teams and coordinating releases and probably several weeks worth of meetings. If you just reuse an old value you can avoid all that since everything is already set up.
There's a good meta-lesson here which is that smart people will do dumb things if you make the smart thing require too much red tape or process.
Processes can block stupidity, but they can also block intelligence if not well designed.
I used to work in HFT. I have seen highly variable practices in this case, including a "mini-knight" incident in the single-digit millions due to tech debt and poor test coverage. However, the most useful change that has resulted from the KCG debacle was adding several layers of kill switches, a dedicated ops team to watch trading and flip the kill switches, and embracing devops automation.
There is a much more serious focus on having a defense in depth, and making sure that problems like this are noticed before they become an issue. Rollbacks are no longer the first action when something goes wrong: the kill switch comes first.
Dead code, tech debt, repurposed flags, and spotty test coverage are everywhere still.
I wouldn’t think of flags as expensive / effortful to make more of, but clearly they must be if people are tempted to reuse them. Can you help me understand what is meant by a flag in this context, and why it would be repurposed?
Repurposing flags not always well-motivated, but one legitimate reason to do this is the memory (and particularly cache) footprint.
Often flags are local to a particular object. If there are lots of such objects, you want each to take as little space as possible. You should check out the contortions linux devs go through to make struct page small [0]. This is important, because there is one such struct per page of physical memory. The memory use is a near-constant percentage of your total memory, and you wouldn't want it to be any larger than necessary.
Even when there are not a lot of these objects, in low-latency software it's important to hit the cache. Your program should always just be as compact in memory as possible.
Semantically flags are booleans (is proposition P true of this object). They are stored compactly as bitsets, often implicitly, say:
This struct will fit into 8 bytes. This is great, as you probably won't waste space to alignment in many cases -- 8 is a good multiple. But if you wanted to add FLAG_9 here, your flags would become a u16, and your struct would, frustratingly, stop fitting into 8 bytes. To avoid this, one might repurpose flags.
Another example of this is intrustive flagging, using, for example, the high or low bits of a pointer aligned to 2^n bytes. If you run out of bits there, not much you can do.
This is pretty much why flags get repurposed. It's also important to mention that things like JSON and protobufs are too expensive for HFT, so you are likely going to be sending structs over the wire. Repurposing flags lets you change a wire format with a lot less friction than adding a byte to a struct. Essentially, it lets you change the minor version number on a protocol and only recompile the endpoints without changing the major version number and recompiling everything.
This is what I thought as well, and I'm in the field. It's legit to try to fit everything in cache.
However, you can still do something about making this safe. For instance the program could do some sort of version check on startup and panic if things weren't correct.
There's a bunch of stuff that needs to be done before the program reaches its low latency steady state, where speed doesn't matter. Might as well add checks there to make sure things are correct.
I remember the Knight Cap event, I was working on order routing at the time.
Things have changed a lot since 2012, and at the same time haven’t. Circuit breakers and position monitoring are no.1 in any sane market making firm. What happened then I can’t imagine happening now (accumulating a huge position for, what was it, 30 minutes? With nobody killing the algos within a couple of minutes?). On the other hand, the perfect world of “code hygiene” and 100% test coverage will never exist in this world, things will slip and they do frequently. What’s better, externally, is the availability of good tools for development and change reviews (bitbucket taking hold, for example), automated deployments, containers, testing frameworks and similar.
This type of software, end to end, is incredibly complex and difficult to reason about when unexpected happens (there was a TTL misconfig for multicast and we never got such and such update? Well, no one thought of that!), esp these days with the influx of ML algos for price generation.
Currently work at an HFT firm. Most of the firms invest well into good DevOps, Trading Systems and SRE teams, to ensure that everything from installing a trading server at the colocation facility, to CI/CD and making changes to the systems configs, is done well. There are also guards in place to ensure that if the system seems to make trades that are way too odd then pull the plug and go down immediately.
Also, any code that does not need to be there, is promptly removed right away.
Where I work at, we have a few people from KCG i.e what was formed after Knight Capital merged with GETCO, after this incident. Sometimes this incident is bought up, although none of them I think ever worked for Knight Capital before this incident.
Some of this is unforgivable, but reflecting on it I also realized that software engineering at quant firms has an almost impossible mandate. You want something akin to the extreme rigor of mission critical software (airplanes, cars, NASA, etc), while also remaining nimble enough to modify strategies as market conditions rapidly evolve.
That truly is scary to me. I can easily* write advanced Solidity and could try to make something big. But I won't, because I know I would not be able to handle the stress and responsibility. One tiny logic error and millions lost. Thanks but no thanks.
*The fact I believe I could easily do it is probably exactly why I'd end up making some huge mistake. ;)
I’m sure that any big contracts are written only by CMM Level 5 organizations using formal methods and provably correct with certification that less than 1 line in a million contains an error. It’s all spelled out in the SOW attached to the RFP.
I don't think I could, since the language was designed by amateurs after no research into safe programming, and is only being improved one gigantic loss at a time.
It is challenging, although, with financial markets, it seems like it would be simpler to have some automatic anomaly detection mechanism to unplug or slow things down to prevent further damage.
There are a lot of preventative measures they could have taken, starting with just not leaving in dead code and paying attention to automated alerting. But the moral of the story is that they got away with it for so long that nobody cared about it anymore. After all, if it were truly a big deal why hadn't it broken years earlier. Then when the technical debt finally got called it bankrupted the entire firm in one go.
Most of us (hopefully) have less devastating technical debt to deal with, but it is still a cautionary tale about what could happen if you ignore it for too long.
The extreme rigor on the one hand seems to require a value judgement of the real benefits to HTF that I'm not willing to make. The remaining nimble'ity, on the other hand, is an odd word to use over agility or old fashioned responsibility. The benefit is proportional to it, but not exclusively.
The rapidly evolving market conditions concern regular trade too. Swift reactions are expected in any other systems application. "almost impossible" is a weasel word. It's almost impossible to win except for the last man standing, is that it? And there's no practical upper limit to nimble'y, though conservative estimates indicate that less work is more.
What's missing is the perverse incentives, corrupt policies, sociopathic leadership, ...
Hard to speak for HFT in general. Like in software, different firms have different levels of hygiene. About half of your bullet points were true of my previous employer, at my time of leaving.
Repurposing feature flags is some kind of next dimension horror for me. We've got quite a few of these to deal with, and if someone started changing what they mean we'd be fucked super fast. Simply suggesting that we alter the meaning of an existing FF would result in the resignation of a non-zero number of project managers on my team.
Rolling back code is another thing I have no tolerance for anymore. The only option we entertain these days is a roll-forward. If your software takes so long to iterate/build that you need to go back to and old version in an emergency, you need to review your languages/tools/frameworks/processes. We maintain a contractual obligation to our customers for same-day code updates (in cases of production/regulatory emergencies) because we have enough confidence in our processes.
You’re likely using human-readable names for the flags and shipping a multi-MB payload of JS and JSON. An HFT firm is likely bit-packing flags so they can send an 8 byte payload rather than 10 and might be using an FPGA hanging right off the PHY to figure out “is this message even interesting to me?”
Your feature flag might be “SHOW_STRIKETHROUGH_PRICING_ON_CROSSSELL_OFFERS”; theirs is a bit mask macro to pick off the 5th bit from the 7th byte. (Why do they care? Because if they allow themselves to get fat and slow, a competitor will take the money.)
Roll-forward only, same day SLA is probably right for your business, but isn’t for a company that could have their systems dusting off $1M every handful of seconds that bad code is running.
Different business problems call for different technical approaches. You should no more adopt theirs than they should yours.
Rolling back isn't about avoiding rebuilds, it's about restoring to a known-good state. Making an emergency patch is typically far riskier than going back to last week's build. We always favor rollbacks, unless there were critical fixes in the current release we absolutely cannot afford to lose.
The only thing worse that comes to mind than repurposing flags is re-using UUIDs, which I have seen some production DB do (no, not mine, thank you very much!).
Reusing the RAM occupied by flags might be do-able in a clean way using guards like the structured enum in Rust, which permits unions (objects that occupy the same space) that always have the right type (i.e., the compiler has knowledge what is in there at each point in time). This mechanism could in theory be extended beyond type-safety to accommodate other contexts in systems programming use cases where memory usage is extremely important.
Rushing out a release without proper software engineering hygiene.
Sounds familiar. From everything listed above, it would sounds like this must have been yet another one of those "Just F-ing push the change the server now, dweeb, the traders are going crazy" environments. That is to say: the real problems were most likely cultural, and not about the sum of a certain set of bad practices.
This is all basic stuff I look to set up in every team, and it's crazy given how these firms work directly with tons of money that they don't have an even higher standard. Guess I wasn't wrong turning down these roles.
Random, but I interviewed at Knight Capital for a software engineering position a few weeks before this all went down. I was in London, so the interview was done over the phone. Picture me in the evening, handwriting C to solve some problem (the fog of time too thick to remember what that problem was), then reading out what I'd written, semicolons and all, to the interviewer. Because of course there was no shared doc. I did very badly. But then, so did they.
Are brackets different in American English? I'd say the Queen should order a second burning down of Washington but, the way things are going, the locals will probably get to it soon.
There seems to be some confusion. I am saying the top is American - at least as I was taught it in school. The bottom is the more typically British variant. But I think the reality is the both America and Britain have variations. To an American ear, bracket to mean parenthesis is not just odd, but wrong. But someone already said they consider bracket to be a catchall term, so obviously this is not universal.
Edit: okay, apparently there is just complete bracket anarchy across America and words have no fixed meaning.
Chiming in as someone who has lived decades in both South East US and North East US. I've also never heard () called "brackets" only "parentheses". This includes in and out of programming circles.
I assumed the comment saying "parens: ()" was talking about US, but the replies seem to be interpreting the opposite.
BODMAS for me too, although the "O" always confused me, the "E" standing for "exponent" always made more sense to me than the O" standing for "[to the power] of"
In my lifetime, “parentheses” has been the canonical way to refer to () in the US. “Brackets” is an umbrella term that can describe one or all of (), {}, and [].
If you’re in a context where only one of the bracket types would normally be used, you can just call them “brackets.” Since many people never really use [] and {}, “brackets” casually refers to ().
To disambiguate, you can call () “open brackets”, [] “square brackets”, and {} “curly brackets.”
And for the cherry on top, {} can be called “curly braces.” That’s my preferred term for them. But it’s awkward to call [] “square braces” and () “open braces” :)
Same. I've seen {} and [] both called braces and brackets, though usually in that respective order. The ambiguity is precisely why they are also called curly braces and square brackets. The double naming is more concrete.
I've never seen or heard any programmer call () brackets or braces. It's always parentheses. I may have heard a math or science teacher call them all brackets or braces, however, since they are synonymous there outside of odd exceptions like interval notation.
I also interviewed at knight a few weeks before everything blew up, didn’t go well, interviewer was one of those guys who finally got to hire someone and figured now it’s his turn to be a jerk. Really smug and arrogant. When I heard how badly things had gone it did bring a smile to my face knowing somewhere that guy was having a Sunday dinner sized serving of humble pie.
Same! Although I interviewed in person in their NYC office. I was very junior at the time and the team I interviewed with was awesome. I (luckily) didn't get the job. I did a few more interviews and accepted an offer from another company where I met my wife!
The interview was incredibly basic compared to modern FANG style full day interviews. The coding portion was basically a fizzbuzz question. They had a one page printout of a Java method and I needed to give the output. I answered it correctly nearly immediately as it was simply testing pass by reference vs pass by value. The main interview was all verbal q&a core java questions.
Before all these code sharing sites were popular, that's how all the phone screens were done. I had to write my code on paper and dictate on phone. this was before 2011-2012
If Yegge's blog is to be believed this is exactly how they used to do it at Amazon too, many moons ago. I think this just used to be common practice. Before my time though.
The incident happened after a technician forgot to copy the new Retail Liquidity Program (RLP) code to one of the eight SMARS computer servers, which was Knight's automated routing system for equity orders. RLP code repurposed a flag that was formerly used to activate an old function known as 'Power Peg'. Power Peg was designed to move stock prices higher and lower in order to verify the behavior of trading algorithms in a controlled environment. Therefore, orders sent with the repurposed flag to the eighth server triggered the defective Power Peg code still present on that server [1]
> Power Peg was designed to move stock prices higher and lower in order to verify the behavior of trading algorithms in a controlled environment.
This is insane. Make one wonder, what is or isn't actually being deployed in prod in 2022.
I actually always think of the Knight case and similar ones when people see a DeFi organization have an issue and extrapolate that to an issue with the entire DeFi concept.
Its so obvious that those people have no clue whats going on in the markets they respect. Truth be told, many of them dont like markets at all. So its just a lack of exposure and compounded ignorance.
Many traditional finance issues are fixable though, there are many more errors which don’t become big stories because they are reasonably reversed as only minor inconveniences.
like how Credit Suisse is going to reverse their Bill Hwang losses? I guess in this conversation we can't distinguish from irreversible asset value and liquidity issue to misdirected transactions that inherit partial reversibility.
similarly, maybe you/they just don't see the headlines of thwarted attacks in DeFi that work specifically due to design considerations.
I'll take the permission to fail. The rapid iteration creates some really fascinating systems in very short time periods, for me. One project implodes, 100 (or 1000) more harden, bigger money comes in creating more assurances for users like easier recovery and compensation paths, all while continuing to rapidly iterate.
Where is this innovation happening? We’ve had crypto for 13 years now, and just yesterday we had people flat out losing 10k into the ether because their transaction “ran out of gas” without completing. Isn’t that plenty of time to develop systems that avoid basic issues like suddenly burning all your money?
I'm generally skeptical of ethereum but i don't think this claim is quite right. First, they didn't lose 10k "into the ether" it just went to miners as a fee for processing their transaction. Additionally, the fee was paid for priority. There is a limited amount of block space so people can pay extra to make sure they're included. I know it's cliche but this is literally a feature not a bug
For your strawman argument, all of their client side wallets told them “its not worth it bro”, the people did it anyway. Thats a consumer issue not a technology issue.
Was just about to comment along these lines. If I read about this a few years ago I would be shocked. Now after seeing so many flubs in the crypto space, my reaction is just 'meh'
Those $440M were lost by rich people who had invested in a hedge fund, not poor people who bought crypto lottery tickets in the hopes of getting rich quick
My small footnote to this story: we had the look on the "blind bid" to purchase the portfolio of erroneous trades on the program trading desk I worked on. Ultimately we didn't bid (thank you risk department) and it traded away to Goldman as the article correctly reports. Would've been a great trade though. I estimated it netted Goldman $2m+
It's paid time off from a financial entity (hedge fund, prop shop, etc.) where you are paid not to compete in their space.
In my case, I had a year to sit-out as a HFT quant from 2010-2011.
Note that it sounds great - you're getting paid not to work - but nobody likes to make you an offer, as it's non-binding and your perceived/real value is declining as the time goes by.
Hey on that topic of a non compete, I've recently been offered a different internal role that would force me to sign a non compete. What was your experience like interviewing/searching for new roles? I would also like to hear what your general opinion on what it make it worth it to sign a non compete, although it sounds like you a generally negative view of them from this comment.
What's amazing to me is that there are many software engineers here who can recognize how errors like this can so easily creep into software (some would say they're inevitable in any sufficiently complex software) but they still somehow think immutable smart contracts on blockchains are somehow still the future.
Really good write-up. Perhaps there's a dawning realization that the model of 'move fast and break things' is fatally flawed?
> "Knight’s IT project managers and CIO should have pushed back on the hyper-aggressive delivery schedule... Thirty days to implement, test, and deploy major changes to an algorithmic trading system that is used to make markets daily worth billions of dollars is impulsive, naive, and reckless."
The fact that since 2008, the portion of all stock trades in the U.S. taking place away from public markets has risen from 15 percent to more than 40 percent is also kind of astonishing. It's long past time to re-erect the walls between commercial and investment banking.
Today there was a 300B oopsie in Europe caused by a Citibank "glitch" bloomberg.com/news/articles/2022-05-02/citi-s-london-trading-desk-behind-rare-european-flash-crash
It only was a sudden drop in share prices, which quickly rebounded. The amount of money that actually changed hands due to that mistake will be tiny in comparison.
> However, a dark cloud remains: market data suggests that 70 percent of U.S. equity trading is now executed by high-frequency trading firms
Is it just me, or was this one of the scarier statements in the article?
High frequency trading does not seem to add anything, it's not a market instrument to balance anything. It's more like gambling with the odds slightly in your favor.
Oh course, I'm a laymen when it comes to high frequency trading and might totally wrong.
Market makers provide cheap liquidity. Or in other words, compare the total transaction costs (bid-ask spread + commission) you would to pay today to trade 100 shares of stock today vs 30 years ago. The difference is staggering.
My experience in quant finance is that strong engineering teams are viewed as cost centers and are not well compensated the way they are at “real” tech companies, especially when the firm isn’t doing well. Management foolishly tends to only value seeking alpha, rather than reliability of existing alpha.
Unsurprisingly poor management tends not to realize when their software is built on a house of cards.
Definitely saw bugs cause large losses (8 digit numbers).
Feel like this way more of the case in the past than it is now. Engineers at top prop shops today are compensated more much generously than at all but the most senior engineers at a FAANG. For example, word is that new grad hires at Jane Street are getting first year guarantees at around the 450k range, which is much higher than an entry level engineer at a FAANG would make.
Culturally, there's some truth in what you say though I think: alpha generators are probably more respected and are pulling more TC in good years.
My brother worked in a medium-sized fintech not long ago. This example is shown in their onboarding process. I guess the same goes for other companies out there.
at the end .. its just money going from one account to another right? Its not like some physical thing that has perished and cant be brought back. Why is it difficult to reverse the transactions?
Knight was probably too messy to rollback cleanly, but that just means it's a matter of cost/complexity/politics... if you're a big enough player, then the exchange will do you favors, like in the LME case.
> The LME announced that all trades will be voided from midnight until 8:15 a.m. on Tuesday when trading stopped and added that it was considering a closure of several days.
> "People will be asking if this really a functioning market... This is meant to be a market of last resort and people can't get inventories to deliver against positions," said Colin Hamilton, managing director of commodities research at BMO Capital Markets.
There’s gonna be a pretty high threshold for this sort of thing. Higher than “one company fucked up and wants a do-over”.
Rules were established after the “flash crash” of May 2010 to govern when trades should be canceled. Knight’s buying binge did not drive up the price of the purchased stocks by more than 30 percent, the cancellation threshold, except for six stocks. Those transactions were reversed. In the other cases, the trades stood.
Exactly this. If you're a market maker, likely your trades impact your own trades too. As you accumulate a position, your average price is going up with it. Trades should not just roll back because one large hedge fund screwed up. Imagine being a retail trader with that expectation. Would be nice!
> Why is it difficult to reverse the transactions?
Why should the transactions be reversed?
If things had gone according to plan, Knight would have made several million dollars that day, some likely because of a mistake by someone else or an unavoidable circumstance, just like it did on other days.
Those other people weren't made whole, so why should Knight be any different?
Didnt the exchanges cancel the trades during the flash crash in 2010. I am sure the buyers of those transactions which were able to buy shares of good companies for pennies would not want to reverse those transactions.
Knight was not a big player so the exchanges had no reason to keep them happy.
>Under stock exchange rules, Knight would have been required to pay for those shares three days later. However, there was no way it could pay, since the trades were unintentional and had no source of funds behind them. The only alternatives were to try to have the trades canceled, or to sell the newly acquired shares the same day.
And then I understand why /r/WallStreeBets and /r/Antiwork is gaining traction.
All it takes is a bit of organisation and the adoption of Govt tactics and practices which is ultimately violence and then just maybe you might see a Govt that works for the people and not the criminals, but I cant picture Bernie Sanders wielding a pitchfork!
Some of the issues mentioned include:
The article suggests improvements that could have prevented the chain of events.For those here who are in HFT circles, have things improved after the Knight Capital Group debacle?
edit: formatting