Compute adds a kind of relentlessness to our intellect; it does not discover new horizons.
Some commenters object to my breezy dismissal of AI doomerist views, the (insane, in my opinion) predictions of 10 percent human extinction or 100 percent over a short time frame.
Now, I grant that AI can create powerful technologies and that the legacy of “technical debt” — poorly designed software on which our grid, military, financial system, and governments rely — is a serious issue. I tend to think AI and the massive compute power needed to run it favors the security side rather than the hackers. But I accept that AI-powered systems run a risk along the same lines as other major industrial accidents.
The commenters ask why they should trust me, when some of the people spreading these views are technical experts. I would argue they aren’t, for the most part. Eliezer Yudkowsky, perhaps the most influential “theorist” in that movement, blogged his way into his worldview. Nick Bostrom, the author of Superintelligence, is a philosopher. Nate Soares is a former Google engineer who wasn’t working on machine learning. He taught himself enough math to write about decision theory and logic, and joined an AI doomer nonprofit. There are some exceptions of course, like Geoffrey Hinton and Dan Hendrycks, though they tend to lean more toward the industrial-accident view rather than out-and-out posthumanism.
But my argument is rather simple. This is not really intelligence, but it is artificial. The chatbots aren’t chatting with you, they are running math over an extremely large corpus of text and spitting it back at you.
The theory is that at some point, AI just takes off, improving itself and pioneering an ever-expanding cone of knowledge that will be incomprehensible to us, and then it will pursue whatever aims we programmed into it or that it programs into itself.
The main problem with the theory of superintelligence, to my mind, is that intelligence is about something. Problems come from ends, and ends come from creatures who need, want, or suffer. Compute has no human problem to solve until we assign it one. The hammer does not lift itself off the bench and search for nails. It must be sent by a human will.
Compute adds a kind of relentlessness to our intellect; it does not discover new horizons. As binoculars help our eyes to see farther than we could normally, AI acts like a prosthesis — a computation system that never tires and runs faster than we can. That is, intelligence is constrained by relevance to human problems the way that natural selection constrains random mutation.
We’ve long had the issue of powerful computer software becoming inscrutable to us. In 2017, two Facebook bots drifted into their own shorthand, communicating with each other endlessly in ways that the creators could not understand. They were shut down because they were useless, not because they were frightening.
We have plenty of experience with machines that are more powerful than us and that can, with bad engineering, or an unexpected context, do dangerous things. Forklifts that get out of control, etc. Just because AI can narrate its “goals” with human language doesn’t mean it has its own willpower underneath. Just as a machine that tears through a chemical plant is not suddenly nefarious because some of the chemicals it combined by accident would be combined by a malefactor aiming at a more dangerous outcome.
What the AI doomers call “misalignment” is not superintelligence, but — no offense — more like super-autism. I use that word because so many in this community of doomers romanticize “neurodivergence” itself as a super-capability. Misalignment means that a powerful machine pursues an end but lacks any way to sense or take into account the unintended social catastrophes it may cause while heedlessly applying intelligence and power to a more narrowly scoped problem. A normal person, alive to those cues, would desist in their intention and make new plans. This is an engineering problem, not an eschatological event for humanity.