The irony is that this is 100% the what MobilEye said in their in their model demo I think 3-4 years ago.
Their CEO said that the regulators cannot regulate all breaking conditions and should should only test for false negatives exactly to prevent this.
He stated that dealing with flase positives is going to be in the best interest of every manufacturer and the easiet thing to do is to disable the breaks completely but that would be fairly easy to detect in any structured test or a test drive.
Any thing else would be a problem because you would be stacking tolerances and worse creating test cases with a clear conflict of interest where the interests of the car maker and the regulator align.
Mostly the message is that regulators shouldn't be investing their resources in checking if the car breaks too often (from false positive in obstacle detection) because the car company has strong incentive to reduce unnessary breaking or driving would be unpleasant and slow for a large portion of the drivers.
The regulators should only test for false negatives, where the car should have stopped, but did not detect the obstacle (false negative), because there, it is a clear threat to safety, and the car company's incentive, while definitely still present, is less pure, as the amount of false negatives is a direct trade off with the quantity of false positives (because it is a treshold: a minimum confidence level from which you decide that there is indeed something in front of the car and you need to break), which make driving more awkward for 99% of drivers
Yep, this is the thing that worries me the most on some of these systems right now. They are going to get rear-ended a lot for the next few years, IMO.
Rear ends have basically no fatality rate though. They do have material costs but if optimizing for no loss of human life it sounds more appealing.
These kind of tradeoffs were things every self-driving car software developer KNEW they were going to have to deal with - the most extreme being the one where the software has to decide who to kill and who to save:
Not sudden random braking, no. The most important advice for driving on slippery surfaces is to avoid sudden braking, and in general to be careful when you brake.
