Couldn't agree more with every point listed. These are fantastic points.
I do not know the author, but reading these points makes me suspect he/she is an experienced software engineer that has been doing this for many years now.
I expect that especially his first point (about being humble in the face of software systems complexity) will provoke many hubris-filled comments. I fully agree with the author: we are incapable of building complex and correct software systems. This is why I am afraid of the hype behind self-driving vehicles.
I have a different expectation of correctness; absolute correctness is impossible, in practice correctness is not a binary condition. Even if something provably executes correctly according to spec, the spec may be wrong (and probably is, since everything interfaces with humans eventually). Everything is in a process of becoming; 100% correctness is not achievable, but it's not desirable either - it costs too much; it just needs to be better than the next best alternative.
The proper goal is robustness.
It is either achieved via clear indication of failure and guidance on alternative solutions - so that the user can handle it.
Or by actually failing gracefully and handling as many error conditions in an always reasonable way.
Usually letting user handle it is more general as long as the failures are rare enough.
Internal correctness is perfectly achievable though. External (correct spec) is not.
There are always some unhandled conditions, due to hardware or external component failures...
Even with 100% correctness, your solution still has the capacity to fail spectacularly.
The universe is infinite in its desire to mess with your expectations and assumptions of what correctness is! Oh, you thought that was 100% correct? Well, have a look: A platypus.
I once worked at a place where a bug in the spec was treated as impossible - if people couldn't get the results they wanted because the spec made it impossible, there was no way to get it fixed. Any attempt to log it as a bug was rejected.
> reading these points makes me suspect he/she is an experienced software engineer that has been doing this for many years now.
And yet, a cursory review indicates they are barely out of high school and has never worked on software in a commercial environment. Not casting stones (I agree with some of what they say too, and what I agree with I find insightful. How's that for confirmation bias?), just observing that this is in fact not based on much experience at all.
autonomous driving is a great example here. It will never be "correct" because there will always be accidents. So the author is right about that. But it is very possible that those cars may be 10x safer than the average human driver some day.
What if you are an above average driver? Maybe for most people it would be an improvement in security, but for you it'd be more dangerous to be in an autonomous car.
If you're an above average driver, a below average driver will ram your car unexpectedly. And there will be nothing you can do, because your reaction times are human.
Also, you're likely not as above average as you think yourself to be.
> If you're an above average driver, a below average driver will ram your car unexpectedly. And there will be nothing you can do, because your reaction times are human.
That doesn't require level 5 self driving cars, only brake assistants.
> Also, you're likely not as above average as you think yourself to be.
That doesn't mean that there aren't above average drivers. I'm not assuming I'm among them, but for them, driving in a self driving car would make matters worse.
It requires more than brake assistants. People are, in the limit, idiots (at least when driving). If you drive long enough, you'll see lots of really hair rising things - and the benefit for the above average driver is that the below-average drivers are off the road.
The benefit to self-driving cars is not only that they are driving (hopefully) better than humans - it's also that they minimize the risk of completely erratic behavior. People don't have accidents when they're doing average. They have accidents when they're at their worst.
Last example I witnessed: At a traffic light, a car going ~30mph suddenly and without warning pulls into oncoming traffic because for whatever reason they thought "must left turn" - despite left turns being clearly disallowed.
That wasn't just a "below average driver", because that doesn't happen every day. That was somebody who was very clearly way off their game. And that's the main source of accidents, doing something clearly wrong that you likely wouldn't even think about on a good day. Or being distracted.
Taking these things out of the equation is way more important than "driving better than the best human on their best day".
And this will benefit the above-average drivers as well. (Also, the average will likely move up, so "above average human" might suddenly not mean a lot)
The idiots you encounter are not all the same people all the time. And sometimes, you're someone else's idiot. You might not even notice when that happens.
All it takes is a moment of confusion, or distraction, or frustration. Or a chemical impairment. Or having a stroke. Or drowsiness.
Braking assistants are good, a self-drive function that has the goal of safely removing the vehicle from traffic--for operators that can suddenly no longer function at speed--would be better. Hitting a panic button and waking up on the shoulder with the hazard lights on is preferable to getting intimate with a pole or guardrail while the cruise control is set to 70 mph.
Our informational machine assistants would also be better with access to vehicle state and sensor data. Your navigation assistant could then tell you, if you are approaching an intersection with your left turn signal on, that left turns are not allowed, before you get there and possibly make a bad snap decision. The nav app on your phone doesn't know that you are signaling left. The GPS signal isn't quite reliable enough to pinpoint which lane you're in. It only knows the positions of vehicles whose operators are currently using the same app.
Obviating snap decisions by drivers should be a goal, because less thinking time means more mistakes. Mobile nav apps have gotten better at this, saying which lane you should be in, and whether destinations are on the left or the right, but they could still do better for in-vehicle improvisation, when the destination changes while enroute, or temporary detours are needed.
Driving ability is almost the perfect case study for Dunning–Kruger effect. Most drivers think they are better than they really are.
So you maybe right in that really good drivers may be safer when in control of the car, but it won't be the case for people who think they are good. And if you leave individual drivers a choice, you can expect to see a lot more of the latter. And, to follow up on groby_b, they are going to ram your car and there will be nothing you can do.
In the end, unless we have a system to reliably select the highest skilled drivers, and making sure they are at the peak of their ability, autonomous cars will be safer for everyone.
> This is why I am afraid of the hype behind self-driving vehicles.
Well, they don't have to be absolutely correct. Just a lot more correct than human drivers across a broad range of driving scenarios to justify replacing human drivers.
I would say he/she is an experienced "person", but many of these points, including some about software, can be understood simply by an honest understanding of human beings.
I do not know the author, but reading these points makes me suspect he/she is an experienced software engineer that has been doing this for many years now.
I expect that especially his first point (about being humble in the face of software systems complexity) will provoke many hubris-filled comments. I fully agree with the author: we are incapable of building complex and correct software systems. This is why I am afraid of the hype behind self-driving vehicles.