A new software release has driven zero miles. You do not know what kind of performance it has -- regressions both happen and are expected with every launch.
The number of miles a release ends up driving is mostly in simulation + some human-supervised on-road testing, and may be low / influenced by the availability of resources ($$$). There are little to no regulations or 3rd party checks on the process used to clear releases.
They do not simulate every release on all X miles driven. They can't, it's super expensive, and that's to speak nothing of challenges like simulation realism or compatibility of the simulator with multiple generations of hardware that isn't backwards compatible.
They could have no deaths for years, then suddenly a whole lot after a bad software push. Recently they issued a recall because of a bug where the car could hit a telephone pole. Imagine if the software had a bug when their cars were ubiquitous on highways, slamming cars into concrete medians? Imagine if a fault could bork critical technologies like the lidar at a high rate across the fleet?
What happens to Waymo if they kill even 1 child due to a bug or lapse in model quality? I imagine many would wake up to the reality of their safety story and demand a way to audit software releases. Or would you continue to let your kid ride Waymo? This is also to say nothing of sensor failures and other ways the car can mess up.
The probability of your dying in a Waymo is not # incidents / # miles, because past performance does not guarantee future performance when it comes to new releases. It's unknown quantity L (likelihood of encountering a dangerous bug per unit time) * T (time spent in car). Without more data about L (which changes based on many factors, like the company's process for certifying releases), L*T could go from 0 to something unacceptably high really fast.
You trust the car you drive today because the parts are quality controlled and regulated. Software drivers are a very new part added to this chain of things to trust. Their quality control process is not yet publicly regulated. The reality is there needs to be an advancement in technology to ensure software quality / safety, ideally from the role of an unbiased auditor that can catch faults and ground the fleet asap, then restore public faith by ensuring gaps in the qualification processes are fixed. No such mechanism exists today.
> They could have no deaths for years, then suddenly a whole lot after a bad software push.
The software release process itself becomes the thing that is tested statistically. If they release 100 versions, and each is safer than the last, it’s silly to think that one can’t be confident in the safety of the 101st without some gov’t software approval/validation process.
> What happens to Waymo if they kill even 1 child due to a bug or lapse in model quality? …would you continue to let your kid ride Waymo?
If they had driven billions of death-less miles and then a software update killed a child, I would obviously let my kid ride Waymo. It wouldn’t be close.
> If they release 100 versions, and each is safer than the last, it’s silly to think that one can’t be confident in the safety of the 101st without some gov’t software approval/validation process.
That's just the thing. Each release is not always safer than the last. Even ascertaining that with any reasonable level of certainty requires a lot of technology. In a company's hands alone, this "trust me bro" amounts to a ticking time bomb since companies have a tendency to trade safety for profit.
> If they had driven billions of death-less miles and then a software update killed a child, I would obviously let my kid ride Waymo. It wouldn’t be close.
You don't think the outcry at Boeing applies to Waymo?
I feel like this concern is mostly a thing having to do with safe rollouts. Just as with any other software, I'm sure there's a gradual rollout to ensure bugs don't just kill a ton of people at once. Google and other large cos have largely solved the reliability problem with web services. while cars are ofc different i think the same learnings apply
> while cars are ofc different i think the same learnings apply
They are very different. Unit tests and stress tests can be conducted on pure software. You can monitor them and if there are failures you can retry.
Robot software needs all the stuff you use for regular software, but then you have this huge space of counterfactuals that can only be verified using large scale simulation. How do you know the simulator is realistic? That it has a sufficient level of scenario coverage? That the simulation was even run? That the car didn't catastrophically regress in a way that will lead to a crash after 1M miles of driving (which in a scaled service will happen quite frequently)?
Oh and a fault can result in a death for Waymo, but not for Google Search. So that's kind of a major difference.
The number of miles a release ends up driving is mostly in simulation + some human-supervised on-road testing, and may be low / influenced by the availability of resources ($$$). There are little to no regulations or 3rd party checks on the process used to clear releases.
They do not simulate every release on all X miles driven. They can't, it's super expensive, and that's to speak nothing of challenges like simulation realism or compatibility of the simulator with multiple generations of hardware that isn't backwards compatible.
They could have no deaths for years, then suddenly a whole lot after a bad software push. Recently they issued a recall because of a bug where the car could hit a telephone pole. Imagine if the software had a bug when their cars were ubiquitous on highways, slamming cars into concrete medians? Imagine if a fault could bork critical technologies like the lidar at a high rate across the fleet?
What happens to Waymo if they kill even 1 child due to a bug or lapse in model quality? I imagine many would wake up to the reality of their safety story and demand a way to audit software releases. Or would you continue to let your kid ride Waymo? This is also to say nothing of sensor failures and other ways the car can mess up.
The probability of your dying in a Waymo is not # incidents / # miles, because past performance does not guarantee future performance when it comes to new releases. It's unknown quantity L (likelihood of encountering a dangerous bug per unit time) * T (time spent in car). Without more data about L (which changes based on many factors, like the company's process for certifying releases), L*T could go from 0 to something unacceptably high really fast.
You trust the car you drive today because the parts are quality controlled and regulated. Software drivers are a very new part added to this chain of things to trust. Their quality control process is not yet publicly regulated. The reality is there needs to be an advancement in technology to ensure software quality / safety, ideally from the role of an unbiased auditor that can catch faults and ground the fleet asap, then restore public faith by ensuring gaps in the qualification processes are fixed. No such mechanism exists today.