"Best suggestion of the article is that we scorn AI researchers who do not have a credible claim that their designs will maintain a basic agreed-on value system after a billion self-managed iterations and upgrades by the AI."
Unfortunately, I don't think that's really going to happen.
When people refuse to fund research in a promising area, others wind up taking up the slack.
Witness America's prohibition on fetal stem-cell research. The US lost its lead in that field when researchers from other countries continued the work.
The same will be much more true of AI research, as a working strong AI would probably be seen as a goose that lays golden eggs.
Of course, it's also a Pandora's Box. So we should be careful. But I don't think scorning researchers in the field is going to be very effective.
Unfortunately, I don't think that's really going to happen.
When people refuse to fund research in a promising area, others wind up taking up the slack.
Witness America's prohibition on fetal stem-cell research. The US lost its lead in that field when researchers from other countries continued the work.
The same will be much more true of AI research, as a working strong AI would probably be seen as a goose that lays golden eggs.
Of course, it's also a Pandora's Box. So we should be careful. But I don't think scorning researchers in the field is going to be very effective.