Iirc it turned out some 20+ year old code was relying on an undocumented/unknown behaviour of dates when they are negative to calculate cashflows on very old perpetual fixed income instruments[1]. If you have a codebase that's large enough and worked on by enough people, for every possible hack/undocumented feature you'll have at least one person who will rely on it in some way.
It cost millions, because the serialization corrupted the USD treasury curve when we wrote it to the database, which meant that even though the code was reverted quickly, for a while no derivatives (like literally no derivatives) in the whole of the trading part of a big IB could price (because essentially all derivatives were priced in dollars using some sort of discounted cash flow mechanism which relies on the USD treasury curve). No price means no risk, because how you calculate risk of a derivative is by finding the underlyers, shocking those underlyers a little bit and calculating the new price (essentially an empirical method of finding the derivative of price with respect to each of those things).
It cost millions, because the serialization corrupted the USD treasury curve when we wrote it to the database, which meant that even though the code was reverted quickly, for a while no derivatives (like literally no derivatives) in the whole of the trading part of a big IB could price (because essentially all derivatives were priced in dollars using some sort of discounted cash flow mechanism which relies on the USD treasury curve). No price means no risk, because how you calculate risk of a derivative is by finding the underlyers, shocking those underlyers a little bit and calculating the new price (essentially an empirical method of finding the derivative of price with respect to each of those things).
[1] https://en.wikipedia.org/wiki/Perpetual_bond