Gavin Newsom Walks the AI Tightrope

O n a single day last week, Gavin Newsom showed that he remains one of the nation’s great political athletes, executing a Summer Olympics–grade threading of the artificial intelligence needle. All on September 9, the California governor attempted to address the public’s anxiety about a future full of man-killing robots while indulging that same public’s depthless affection for instant answers to life’s deepest questions (e.g., “What would Shakespeare call a leaf blower?” “A tempest-bellows”).

Newsom first signed two “first-in-the-nation” AI regulations; one creating a state registry and standards for AI auditors, the other to regulate and register private AI auditors in California. And then, with no apparent sense of irony — but perhaps with the very same gubernatorial pen (reportedly a Montblanc StarWalker Metal) — Newsom announced the creation of AskCA, California’s official artificial intelligence assistant.

In Newsom’s California, artificial intelligence is apparently dangerous enough to require an expanding apparatus of government regulation but trustworthy enough to answer questions about government itself.

Newsom has attempted to ride the AI dragon for three years, always emphasizing its dangers. He began with a 2023 executive order directing the state to develop standards for “ethical, transparent, and trustworthy AI.” In 2024, he signed a hand-cramping 17 AI bills in 30 days, a “nation-leading” package regulating deepfakes, watermarking, and the treatment of children and workers. Last year, he signed SB 53, what his office called the nation’s strongest frontier-AI safety law. Last week, he approved 13 more bills governing chatbots, children’s privacy, and minors’ access to addictive social-media features, again advertising his artisanal handiwork as a national model.

AskCA is different, like an escape from the governor’s panic room. It promises to guide Californians through the state’s hundreds of agencies, departments, programs, and websites. Built by the California Office of Data and Innovation using Anthropic’s Claude model, AskCA will search official state, federal, and county sources, explain complicated subjects in plain language (and in English and Español!), and provide direct links (“enlaces directos”). That’s a worthy undertaking. Anyone who has ever attempted to navigate California’s unemployment system, for instance, might willingly consult a Magic 8 Ball instead.

AskCA isn’t yet generally available. Its website invites Californians to sign up for pilot testing and says participants will be able to explore the chatbot in October. Newsom nevertheless says the tool will provide “trusted” resources and asserts that its answers are “backed by verified sources.” Elsewhere on the AskCA website, California warns that its AI can make mistakes and may provide information that is “wrong, out of date, or not right for your specific situation”— in which case, users are admonished to check with the responsible agency. The state also disclaims liability for errors and omissions and says it isn’t responsible for an “unexpected answer.”

The unanswered question is who determines what AskCA considers trusted and verified. Anthropic itself has already shown a willingness to involve a self-interested organization in shaping the rules and training surrounding a specialized Claude product. In developing Claude for Teachers, Anthropic worked with the American Federation of Teachers, the union headed by Randi Weingarten and currently representing roughly 1.9 million members. The firm even tapped Weingarten for an advertising endorsement that echoes the effusive tones of Newsom’s own communications: “We’ve been working with Anthropic on a Gold Standard that sets out industry best practices for safety and privacy in K-12 education,” Weingarten said in Anthropic’s press release. “It’s important that Anthropic is committing to these principles in their new Claude for Teachers — a tool designed by and for educators to assist them instructionally and hopefully give them more time for the human relationships at the heart of learning.”

That doesn’t establish that AFT trained Claude’s underlying model, much less that it has influenced AskCA. But it does show Anthropic’s willingness to admit partisan partners into R&D. A teachers’ union is no neutral authority on school choice, collective bargaining, teacher discipline, or parental rights. So, when California says “subject matter experts from across state government” are “continually” helping inform AskCA’s responses, users should brace for impact. The state’s published description doesn’t identify those experts or say who selected them. Nor does it explain whether competing viewpoints are represented or provide detailed rules governing the selection and ranking of sources from which AskCA constructs its answers.

AskCA’s privacy promises deserve similar scrutiny. Its public-facing “About” page says California “will not share or sell your data,” will not train the model on users’ conversations, and removes personal information from conversation logs. But detailed privacy materials say something different: Conversations are saved, Anthropic “collects and stores” users’ data, and the AskCA team reviews conversations to identify gaps and errors. Information may also be forwarded to employees in other state departments or disclosed to state or federal agencies and law enforcement when authorized or required by law.

AskCA claims that Big Brother will scrub personal details, including names and Social Security numbers, before conversations are saved for “long-term review.” The same policy says the conversation is collected and saved during the chat before describing that subsequent scrubbing for long-term review. The pilot’s research-participant notice says some research information may be retained for as long as two years, but the published AskCA materials do not specify how long Anthropic retains the original prompts, whether Anthropic personnel or subcontractors can inspect them, or whether a deletion request necessarily reaches backups, diagnostic logs, or copies already forwarded elsewhere.

Automated redaction also cannot by itself eliminate the possibility of reidentification: The National Institute of Standards and Technology warns that de-identified information can sometimes be reidentified and that merely masking or removing identifiers may not provide sufficient protection. “I was dismissed by this school district in June” or “my restaurant at this address burned in the Palisades fire” may identify someone without a name. Combine a few details — an occupation, ZIP code, medical condition, government program, and date — and an ostensibly anonymous conversation can acquire a face.

Newsom’s administration should release AskCA’s data-flow map, Anthropic contract, retention schedule, source-selection rules, reviewer roster, and independent assessments before inviting Californians to describe job, disaster, and family problems to the system. AskCA explicitly presents those kinds of life events as intended uses of the service.

California insists that AI companies seeking to do business with the state demonstrate responsible policies and meet rigorous privacy and security standards, while its new laws impose additional auditing, transparency, and accountability requirements on private AI systems. Its own chatbot should meet the standard Newsom is so eager to establish for everyone else.