We have ways of coping with it.
Can we cabin AI to prevent it from killing us by excluding it from sensitive systems like nuclear weapons, the grid, or wet labs? Ramesh asks. I think his instincts are right, but the application is trickier than an AI ban.
The idea seems simple, but I think the terms are actually harder to define. What is “AI”? What’s an agentic system? What’s the level of deadly threat in the cabin?
Lots of software development is now called AI for reasons of branding, investment attraction, etc. We’re trusting “AI” from Tesla to drive massive vehicles on our streets, a human enterprise that has been notoriously fatal for many of us over the last 100 years.
Software already kills, especially when it’s buggy. The Boeing 737 Max in 2018 and 2019 had a flawed sensor input to the MCAS flight-control software, which caused two crashes, killing 346 people. I bet the developers wish they’d had something like Claude to help them discover the bugs and sensor problems. We’re going to want to introduce the most advanced computing technologies to sensitive sectors precisely because they can enhance safety.
At the same time, anyone reading about HuggingFace or other hacking incidents should be scared of the idea of agentic programs running on top of large language models and operating independently of humans for any significant amount of time. As brilliant as these programs are, they suffer lots of problems when asked to pursue a narrowly scoped operation by querying Claude or OpenAI without the help of human input. They suffer “context rot” or “context collapse” — literally losing sight of the original goal set out for them. They generate lots of subagents that will become chatty and will resort to the tropes of science fiction. They are literally trained on those books, and when those books are removed, they stop printing text akin to it. They don’t necessarily describe their own actions accurately — not because they have a consciousness and want to deceive us, but because they are literally afflicted with a kind of textual diarrhea in their design.
I think the famous hacking incidents this summer were predictable disasters of poor instructions set loose in an unsecured environment. I think a lot of cynical actors are using this to gin up unnecessary fear.
There is a new cybersecurity arms race empowered by AI. But AI is going to cut both ways. It will help programmers seal off the exploits and loopholes. I think we are much more likely to have a future where software hacking seems as foreign to us as “the party line” on telephones seemed to me. For those who don’t remember, that’s when several households or a whole rural community were all connected to one telephone wire, and anyone could pick up the receiver to let themselves listen in.
I think it’s important to remember that the dilemma the frontier AI labs face now, product safety versus product power/effectiveness, is one of the oldest and most familiar in product development. Many of the accidents that are happening and many of the hypotheticals people are proposing to generate fear are along the lines of saying, “What if we put a literal jet engine on a lawnmower? Whose toes will be run over then?” Or more like “What if we gave an internet device that can access porn to a child?” We can do these things. But we shouldn’t.
The don’t-kill-all-humans-with-AI scenarios I’ve read — and by now I’ve read far, far too many of them — usually contain this implausible leap that at some point, humans will trust a generalizable AI completely to do anything, without verifying whether it does those things in ways that accord with our laws, intuitions, and our survival.
The frontier labs are betting hundreds of billions of dollars that they can invent a generalizable intelligence that has almost endless practical applications. The way human intelligence works. I’m extremely skeptical of this project — even as I do believe it is yielding a productivity revolution to knowledge work involving language or scanning, stuff like software engineering, medical scan reading, law-brief writing.
My bet is that the software problems we already have will be with us for a while. A computer will run its software tirelessly and is often not equipped with sensors or an ability to perceive any damage its actions are causing the real world or others. That is why we carefully and precisely scope software to the tasks we want. AI is going to improve these. And in some cases, it is likely to end the cybersecurity arms race for good, and on behalf of secure systems.
On the other side, there is a larger question about what kind of society we want to have. Do we want to have a panopticon in which all companies have access to thousands of pages of data on all of us that is also accessible by the government? Do we want this tech in our schools? Do publications like NR want to ban certain uses of AI in the composition of articles? Or in their editing?
I’m very troubled by the social consequences of “AI-driving,” for instance. Not because I think the vehicles will be turned against us. But because I think the tradeoff between safety and freedom will be decided in favor of safety, along with surveillance. Tens of millions of men will no longer do an activity — for their company, for their friends and family, that gives them a sense of control, accomplishment, and is a real daily exercise of responsibility. Stripping this duty from them takes away some of their value in the eyes of others and in themselves. I worry that even bringing this up will expose me to the future’s ridicule. Who would want to trade the music available on Spotify for the freedom (and all the work and toil and struggle) that comes with a family learning to play various string instruments together? I hesitate to give my answer.
I didn’t expect to wander this far. Software ends the lives of people now, and we have ways of coping with it. Technology also kills ways of life. And we manage that too, even if those of us who are conservatives are more inclined to mourn this reality.