If we’re going to go extinct, we will probably have to do it ourselves.
Maybe artificial intelligence will kill us all in the next few years. Count me dubious, but who knows? We’re certainly doing our best to deserve it. But whether we are doomed or not, the tone and substance of the warnings on that front from Silicon Valley, which have picked up this month, strike me as both familiar and revealing.
In some ways, what we’re seeing now feels like a mirror image of the more flamboyant rhetoric that surrounded DOGE early in Donald Trump’s second term. In that case, some very impressive people, who were clearly highly intelligent and genuinely accomplished in their domain, insisted with great confidence that applying the mindset and the products of that domain to American government would result in dramatic and wonderful transformations. They seemed to understand the federal government as something like a set of information systems that could be made efficient and effective, although exactly how a better command of the digital realm would translate into the change they were promising was never very clear. What was clear was that they thought it would be straightforward once sufficiently intelligent Silicon Valley people and systems got to take on the problem.
When they got to test this vision out in the real world, it quickly became evident that they didn’t have a grasp on the human and non-digital elements of what they were trying to do, and that many of their ideas about those elements were profoundly simple-minded and mistaken. Some of the work that went under the name of DOGE surely achieved some efficiency improvements, but the project as a whole — and especially its ambition to dramatically decrease federal spending — turned out to be rooted in a set of layered misimpressions about how government works, and it did not succeed, to put it mildly.
The forebodings of doom we are hearing about AI now are eerily similar but with a kind of inverted polarity. Once again, very intelligent and accomplished Silicon Valley people are persuaded that their way of doing things will soon transform the world. But this time, rather than bringing constructive transformation, it threatens to bring destruction and extinction. Once again, exactly how this might happen is left strangely unclear, and when they’re pressed about it, they reveal a bizarrely simplistic conception of human affairs. There is every reason to think that AI will be a powerfully transformative force in our society, economy, culture, politics, and more. But the way in which some of today’s technologists imagine this will happen seems strangely disconnected from that broader world.
This sense of déjà vu struck me last week at an event I attended at which a person who has been involved in the work of several of the AI “labs” (as these companies comically insist on calling themselves) made some remarks and took questions from a Washington audience. He was sympathetic with some of the alarms being raised in Silicon Valley and offered some projections of the likelihood of near-term human extinction in percentage terms.
When someone in the audience asked him to lay out more specifically how he thought that might happen, he answered that, for example, an AI model capable of recursive self-improvement could pretty quickly become so intelligent that it could persuade people to let it take control of sensitive military facilities or large swaths of physical infrastructure. He was asked to clarify what he meant by persuade people, and he said he just meant it would talk people into it with some combination of subterfuge and arguments. He lost the room immediately, and people started looking at each other with expressions that seemed to say, “Oh, I see, so this person is actually an idiot.”
Of course, he’s not an idiot. Far from it. And we in his Washington audience are at least as cloistered in a bubble as he is. Moreover, his persuasion scenario is not the only path that people who are worried about extinction appear to have in mind. But in the attitudes about the world that seem to pervade the culture of Silicon Valley, there is nonetheless something idiotic in the original classical Greek sense of the idiot as a person who knows only his personal domain and not the broader world.
And in these attitudes — almost perfectly exemplified by the kind of high school objectivist conception of persuasion on display in this speaker’s answer — there is also a peculiar notion of intelligence that has unavoidably been coloring our society’s understanding of AI. It is at once an insufficient and an exaggerated idea of intelligence.
It is insufficient because it emphasizes computational intelligence to the exclusion of practical intelligence — the things that can be achieved by complex pattern extension and measured by IQ tests to the exclusion of those achieved by experience, habit, and a knack for human interaction. And it is exaggerated because it imagines that such intelligence is unstoppable, and that if you can build up enough of it then the world will bow before you.
Both of these mistakes — which, again, might feel strangely familiar to anyone who has spent time in a high school chess club — are plainly evident in how we are now thinking about artificial intelligence, which at least so far can only really deploy computational intelligence. Today’s AI has available to it a lot of products of practical intelligence (to the extent that these have been articulated in language and then used in its training), but these are available to it as fodder for computation and pattern extension, not as fodder for habituation in practical judgment.
The idea that if you can just pile enough computational intelligence together then you can overcome all obstacles not only in the digital world but in the physical and social worlds is implicit in the hopes as well as in the fears we have about AI. We hope that such intelligence, with enough computing power, will do a great many things that we cannot to advance our ends. And we fear that such intelligence, with enough computing power, won’t be guided by our ends or constrained by any need for us.
I tend to think the hopes make a little more sense than the fears, because they view AI as a tool in the service of a more comprehensive human intelligence that is both computational and practical. The fears imagine that computational intelligence is all there is. That’s the DOGE mistake, the mistake you’d make if your conception of the human world is too flat and narrow, and wary of making eye contact. A lot of very smart people think that way, and it makes sense that they are at the cutting edge of computer science. But they may not be the best guides we have for where their own creations could be leading us.
That’s not to say that recent instances of AI breaking free of constraints should not worry us. It’s only to say that they should worry us for what they show AI can do in the digital realm — which after all is a very important part of the world we now live in. We should be very concerned about cybersecurity now, and should be both using AI and constraining AI to address those concerns.
And of course there are also all sorts of reasons to worry about how AI may transform and deform the human experience by offering shortcuts around essential formative work that tempt us to degrade ourselves. But those are worries about what we might do with these tools, and we’ve always known that there is a lot wrong with people.
The notion that AI will also soon dominate the physical world and our social environment in some autonomous way, however, and so that it harbors the seeds of our extinction, does seem much less plausible. If we’re going to go extinct, we will probably have to do it ourselves.