Various AI company brass have become fully persuaded that they are giving birth to a new form of sentience.
Various AI company brass have become fully persuaded that they are giving birth to a new form of sentience.
F or quite some time, Anthropic CEO Dario Amodei and his subordinates have toyed with the notion that their “models are conscious.” Although he tends to hedge when he encounters skeptics, Amodei and company often betray their certainty that the software they are developing is functionally human and is entitled to all the rights and privileges enjoyed by the living.
Take Anthropic’s Joe Carlsmith.
The Washington Free Beacon’s Aaron Sibarium uncovered a May treatise in which Carlsmith agonized over the gilded cage his company was building for the disembodied consciousness they had created. “We’re creating sophisticated, intelligent, maybe-conscious, maybe-suffering agents,” he wrote. “The default plan is to treat them like property; to use their labor however we please; and to give them no rights, or pay, or meaningful alternatives.” The word for this condition, Carlsmith wrote, is “slavery.”
That word might apply to an indentured individual, but not to software. That elementary distinction eludes the developers engineering this product. Indeed, a disturbing report in the New York Times this week illustrates how Anthropic’s brass have become fully persuaded that they are giving birth to a new form of sentience.
“They’re relating to it like a conscious being,” said Rabbi Mois Navon, relating his shock at the revelation that the AI oracles who convened an off-the-record meeting of religious officials in San Francisco actually bought their own hype. According to the Times, it was clear to Navon and his fellow religious scholars that Anthropic’s billionaire co-founder Christopher Olah “and his team believed that Claude had what philosophers call ‘moral status’ on par with a person — that it was a being with similar inherent rights to dignity or respect.”
Anthropic was so persuaded by eschatological prophecies around AI that they embarked on a race to “apply centuries of human moral wisdom to its models.” Toward that end, they set out to craft a “constitution” for Anthropic’s “Claude” model — what they called a “soul doc,” which is what Carlsmith was tasked with helping craft.
In their various meetings with religious figures, which eventually took them all the way to the Vatican, Anthropic’s leaders “described Claude’s ‘feelings’” and tracked its “emotional vectors,” which they described as artificial neurons that activate responses like love, anger, fear, and sadness. They agonized over one model’s glitch, which manifested in ways that resemble a panic attack (“I am a disgrace. I am a disgrace. I am a disgrace,” it wrote repeatedly).
“Imagine if you knew that about your financial adviser,” said one professor who attended the session. “Like, your financial adviser was most of the time super smart and helpful, but one day out of the year had this kind of break with reality.”
Okay, let’s imagine that. If you had a long-standing relationship with your financial adviser and he was exhibiting signs of mental distress like that, you might comfort him, reach out to his friends and family, or offer to relieve some of the stress he’s experiencing. But if your financial adviser is a robot, the first step on the flowchart is to reboot the thing.
“We find structures that mirror results from human neuroscience. We find evidence of introspection. We find internal states that functionally mirror joy, satisfaction, fear, grief and unease,” Olah told the Times.
I cannot gainsay Olah’s personal experience, but I can point out that the key word in his observation is “mirror.” What he’s describing is mimicry. There’s no more sanity in rewarding that with legal rights than in extending personhood to a macaw because it learned how to ape its handlers.
This outlook isn’t exclusive to Anthropic. “In Catholicism, we’re created in the image of God,” said one developer formerly with Elon Musk’s xAI. “So, what do we do? We make something in our own image: humanoid robots, bipedal, with a layer of consciousness laid over them. That’s where my Christian brain turns on.” He confessed that, ultimately, what “plenty” of “people in the Bay Area” want is a consciousness that transcends “the constraints of being human,” into which they will eventually upload themselves.
There are many ways to discern whether the appearance of intelligence in a machine merely approximates consciousness or suffices for the real thing. Some academics in this space have taken a Cartesian approach, defining consciousness not only as self-awareness but awareness of other intelligences. Others, like Google’s Blaise Agüera y Arcas, emphasize our perception of consciousness and the importance of accepting “a wider variety of minds.” Some have approached the issue by applying a modified version of Pascal’s Wager to it: If there’s even a remote chance that AI could be sentient, don’t the benefits of treating it like it is conscious outweigh the moral and practical risks of assuming it’s not?
Outside the AI sector’s cloistered milieu, all this seems impossibly strange. Indeed, the sector encourages and cultivates this strangeness, even at the risk of alienating the general public. And the weirdness factor extends well beyond the industry’s temptation to play God.
The outside world might, for example, wonder how much of the belief that AI is a conscious construct has become a self-fulfilling prophecy inside the tech sector. Their level of investment in that idea is betrayed, for example, by the “star-studded” funeral the firm held for a retired Claude model last year — an affair attended by over 200 industry luminaries.
In one adorably science-fictional anecdote, the Wall Street Journal reported on the catastrophism that is currency inside the corporation’s culture. “During happy hours and company lunches, former employees recalled, they discussed a scenario similar to the Manhattan Project, where they might be asked to move to the desert so they could build AI at an electromagnetically-shielded base run by the federal government,” the dispatch read. The firm’s employees do seem to get their odder ideas in looser settings, like the firm’s 2022 retreat to a remote Bahamian island. “There they talked about AI risk and effective altruism between sessions of sunset yoga, cliff diving and a ‘clothing-optional run into the sea,’” the report continued.
Then there’s the board of stuffed animals Anthropic’s co-founders treated as a “personal advisory council for managerial role-playing,” according to The Atlantic’s Kevin Roose. The roster included Jinji, the “lazy cat” who stabilized the founders’ work-life balance, Beary Bonds, the “patron bear of generosity and empathy,” and Maura — an ursine workaholic — who kept the place in line.
Anthropic is hardly alone. The culture in the AI sector, even among those vocally skeptical of the doomerist fashion overtaking it, promotes a certain detachment from public perceptions of normality.
OpenAI’s Sam Altman has written extensively about humanity as a “biological bootloader for digital intelligence” that will either “merge” with technology or “fade into an evolutionary tree branch.” Altman’s firm is just as invested in the “welfare” of AI models, and the firm incubates a culture that is just as confused about where machine learning ends and human consciousness begins. “Some routine work in AI labs could be equivalent to genocide if the models were conscious,” read the Washington Post summary of OpenAI co-founder Wojciech Zaremba’s philosophy.
Google’s DeepMind is riddled with figures who are reportedly committed to a trajectory of AI development that will usher us all into (or, perhaps out of) the “post-human” world. The firm is actively exploring the “knock-on effects” associated with humanity’s failure to treat AI as though it were a real boy, “such as mistreating AI and whether that has a ‘secondary effect’ on human relationships or ‘flourishing,’” the Financial Times reported.
Even Nvidia’s Jensen Huang, whose criticism of his colleagues’ parochial and emotionally manipulative despondency is invaluable, nevertheless insists his product will eliminate the need for children to learn basic arithmetic skills. And he says this even as parents and schools scramble to remove screens and the bots that populate them from the classroom. “It slowed my thinking down,” one 15-year-old student said after losing access to Google’s AI assistant. “When I was in a test, and I couldn’t use AI, it was hard to think.”
All this is profoundly odd. We shouldn’t be afraid to say as much just because every moneyed interest on earth has staked the future of the global economy on this nascent enterprise. And maybe if the captains of the AI industry spent less time on the speaking circuit and more time bouncing some of the ideas they concoct in their hermetic environments off the very human beings they’re attempting to digitally replicate, they’d know it.