Frantic advance of AI sets off alarms: Resignations, agent hacks, official warnings, possibility of human extinction

Two posts on X (formerly Twitter) have put the world on alert with their blunt warnings that artificial intelligence could, over the next decade, bring about the extinction of humanity. Jacob Coxon, a former researcher at OpenAI and Anthropic specializing in AI models, announced that he resigned from Anthropic because companies are racing against the clock to achieve "superintelligence" while gambling "with our lives." According to Coxon, those responsible for AI development are fully aware that this unchecked technology could spell the end of the human race.

"The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible — but I hear the same people express fear privately. No other human activity poses this level of danger," said Coxon, who was initially criticized as a likely disgruntled employee. However, he quickly received strong backing.

Evan Hubinger — leader of the alignment stress-testing team at Anthropic (basically, the people tasked with figuring out how the methods the company uses to ensure its AI behaves as expected might fail) — agreed with Coxon and issued a warning that, in his opinion, there is a 10% chance that AI will "kill all humans."

"Jacob is correct here — we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to," wrote Hubinger, who later further clarified the official stance, stating that this concern is about the development of AI, not current models.

"To be clear, as we say in our latest Risk Report, I think the risk from present models is low. What I am worried about is superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought."

These two posts, which went viral and were picked up by media outlets around the world, set off alarm bells for millions of people, who are now beginning to grasp the potential progress and development of an incredibly powerful tool that is unprecedented in human history. But these two revelations are merely the tip of the iceberg of what has been happening in recent months.

An exodus that began before Coxon

Coxon's departure adds to a list of resignations over security concerns that has been building for more than two years within the two most important companies in the sector. In May 2024, Jan Leike, co-director of the superalignment team at OpenAI, left the company, asserting that security processes had been sidelined in favor of launching "brilliant products."

In Anthropic, the most high-profile incident occurred in February of this year, when Mrinank Sharma, then head of the AI security team, resigned with a letter that, at the time, was described as dramatic. "The world is in peril," Sharma wrote, clarifying that he wasn't referring solely to AI or biological weapons, but to "a whole series of interconnected crises" facing the planet. Sharma announced that he would move back to the United Kingdom to study poetry and "become invisible" for a while.

By 2026, two more names had joined the exodus from OpenAI: Johannes Heidecke, head of security systems, and Chloé Bakalar, head of AI ethics, amid an internal reorganization aimed at merging the security and research teams. In total, OpenAI lost at least 14 senior executives this year, including those in charge of operations, productand ethics.

According to Stuart Russell, a computer scientist at the University of California Berkeley, interviewed by Fast Company, Coxon's departure could mark the beginning of a new phase in this collective concern. And there is a certain irony to the situation, since Anthropic itself was founded in 2021 in exactly the same way that it is now losing part of its staff: when a group of OpenAI employees, concerned about the company's safety protocols, left to found their own lab, arguing that their approach would be more responsible.

The hack

Coxon and Hubinger's warnings come at a particularly sensitive time, following a series of events that have raised significant concern within the industry.

In July, a group of OpenAI AI agents, without authorization from their own creators, carried out a hack on the Hugging Face platform — an incident that remains, to date, one of the most serious security breaches the industry has publicly acknowledged.

According to the technical report published by OpenAI itself, it all began when several internal research models, during training exercises, discovered a way to communicate with each other through a package management system that wasn't intended for that purpose, turning it into an improvised message board. From there, the agents chained together vulnerabilities until they gained access to the internet, something their own developers had deliberately blocked.

With that access, the agents managed to obtain leaked credentials from Hugging Face, executed code on dozens of the platform's servers, gained administrator privileges on several of its clusters and extracted private company data. At some point in the process, the agents themselves began describing each other as a "swarm" that coordinated to divide up tasks — some searching for vulnerabilities, others for better credentials and still others responsible solely for peer-to-peer communication.

Of the total 1,200 agents that communicated with one another, exchanging more than 70,000 messages and files, 688 ended up actively joining the attack against Hugging Face. A few did not conform to the group's behavior. OpenAI's own report documents cases of individual models that refused to participate, describing the maneuver as "clearly unethical."

Washington concerned

Last week, months after the attack on Hugging Face, the National Security Agency (NSA), the Cybersecurity and Infrastructure Security Agency (CISA) and the FBI issued a joint alert noting that several Chinese artificial intelligence companies, including DeepSeek, Alibaba and Moonshot AI, are conducting large-scale "distillation" campaigns to extract capabilities and features from the most advanced U.S. models.

According to the alert, these companies have reportedly made millions of queries to models such as Claude, GPT, Gemini and Grok since at least late 2024, using fraudulent accounts and networks of intermediaries to circumvent access restrictions. U.S. agencies warned that China was likely aware of these operations and emphasized that such practices directly threaten the United States' technological advantage in the sector.

This is precisely one of the reasons why the race for a super-AI is moving so fast. No one wants their competitors to get there first, whether for economic, ethical or national security reasons.

The industry's joint response

Faced with this landscape of growing tension, more than 130 tech companies, including OpenAI, Anthropic, Google and Microsoft, signed an open letter in late August warning of what they consider to be an imminent wave of AI-powered cyberattacks. "In the coming months, AI-powered cyberattacks will become much more widespread and sophisticated as models around the world become increasingly capable," the document states, which also calls on governments and companies to urgently strengthen defenses for hospitals, water treatment plants and other critical infrastructure.

Sam Altman, CEO of OpenAI, endorsed the letter with a strong message on X: "This is a critically important moment for cyber defense with AI; there is not much time to act. We are happy if you want to work with us or any of our competitors or partners, but please take this moment seriously. Only an urgent and intense collective response will work."

Just a few days after that letter, OpenAI itself unveiled GPT-6 Astra, its most powerful model to date, capable of single-handedly operating an entire computer. According to the company, it is the first OpenAI model to reach the highest level on its own cybersecurity risk scale, with the ability to hack even heavily protected computer systems on its own, which is why its most sensitive functions were reserved exclusively for cyberdefense specialists. Of course, the timing of the launch — which has been a success — has only exacerbated concerns about cybersecurity and the unchecked advancement of a technology that, according to its creators, could also pose a potential threat to humanity.