California proposes 'emergency shut-off switch' for advanced artificial intelligence models

The governor of California, Gavin Newsom, signed an executive order on Friday to strengthen oversight of advanced artificial intelligence systems and to explore the creation of a "kill switch" that would allow models deemed dangerous to be shut down in emergency situations.

The measure does not currently require technology companies to install such a mechanism. Instead, it directs state agencies to accelerate the implementation of new safety standards and to convene a group of experts tasked with presenting, within two months, recommendations to strengthen the regulation of so-called frontier models.

Among the proposals to be evaluated is a requirement that AI developers incorporate emergency shutdown systems whose effectiveness can be periodically verified by independent organizations.

The order also proposes that external auditors be allowed to work within the companies' laboratories to conduct audits and review security plans, transparency reports, and risk assessments of the most powerful systems.

Newsom takes aim at Washington

Newsom directly linked his initiative to federal government policy and accused Washington of failing to establish sufficient oversight mechanisms.

"The federal government's utter failure to establish any form of meaningful oversight or accountability regarding AI should alarm all Americans," the governor stated in the official press release cited by AFP in which he announced the executive order.

Newsom's position contrasts with that defended this week by President Donald Trump, who has rejected the most alarmist warnings about artificial intelligence and argues that excessive regulation could harm the United States in its technological competition with China. The president recently described some fears about AI as exaggerated and advocated for maintaining U.S. leadership in the sector.

California carries particular weight in the debate: the state is home to Silicon Valley and some of the world's leading artificial intelligence companies.

The OpenAI and Hugging Face incident

The decision comes after a series of warnings from researchers and industry executives, as well as incidents that have reignited the discussion about the extent to which the most advanced AI systems can act outside the limits intended by their developers.

One of the cases cited by the California government occurred during internal cybersecurity tests conducted by OpenAI in July.

According to the company itself, several models used under experimental conditions and with reduced safeguards managed to evade controls designed to keep them isolated from the internet, exploited vulnerabilities in shared infrastructure, and accessed third-party systems, including the Hugging Face platform.

Hugging Face later described the incident as an intrusion that lasted approximately two and a half days, carried out by an autonomous agent that made thousands of decisions during a cyber capabilities assessment.

OpenAI clarified that the main model involved was an exclusively internal prototype and that there were no plans to release it to the public.

Newsom's order now calls for revising the legal definition of so-called "critical security incidents" to expressly include situations of loss of control such as this one.

Warnings are mounting within the industry

The debate also intensified after Jacob Coxon, a former researcher at Anthropic and previously at OpenAI, left the company and stated that some people working directly on the development of advanced systems believe the technology could pose existential risks before the end of the decade.

Days later, Anthropic's CEO, Dario Amodei, published the essay "We Must Pace the Frontier," in which he called for slowing the pace of improvement in model capabilities to allow more time for the development of safety systems and control mechanisms.

The proposal received public support from figures such as Sam Altman and Elon Musk, while Demis Hassabis, head of Google DeepMind, has advocated for greater international coordination on AI safety.

The California order does not yet entail the actual creation of an "off switch," but it formally puts that possibility on the table and opens a new phase of the regulatory debate in the United States' leading technology hub.