"I am afraid that this will have a chilling effect against innovation in this area" is the sort of way that software engineers respond to similar problems in their space.
We need to hold ourselves to at least this good a standard.
With notable exceptions (e.g. Therac-25, or self-driving cars) most of us have the luxury of working in spaces where mistakes may be annoying or costly, but not deadly.
I agree. When working on products that have the potential to directly maim or kill people, we should involve domain experts and subject them to rigorous testing.
We need to hold ourselves to at least this good a standard.