Megan McArdle raises an idea that has occurred to me too, and thereby makes me think there might be something to it:
We should limit AI access to any means of causing mass death: nuclear command and control, materials and equipment needed to make biological or chemical weapons, and critical water and power infrastructure. Call it the “murderbot principle.” We should try, as much as possible, to keep AI away from anything that could be used to kill large numbers of humans — keep AI out of devices and facilities, segregate them from the internet and lock down supply chains. . . .
Hostile AI programs are going to be a problem no matter what we do, as will hostile humans empowered by AI. But as long as humans retain control of the physical world, AI can’t actually kill us all.
I’d be interested in hearing the counterargument.