AI Insiders and the UN Warn of Extinction Risk as Trump Races Ahead

There have been a lot of headlines raising concerns as the race for AI continues. Just seven days ago, the most talked-about AI product was a lifestyle assistant that could book tennis courts and order takeaways. By the end of the week, the UN human rights chief Volker Türk had issued an open letter warning about existential risks, three of the world’s biggest AI executives had called for a slowdown, and US President Donald Trump had told them to keep racing. So, EM360Tech decided to break down what happened, who said what, and why the disagreement matters.

What Triggered The AI Safety Debate

On 8 September 2026, Jacob Coxon resigned from Anthropic after roughly three years of pretraining research split between OpenAI and Anthropic. In a post on X that drew more than 90 million views within a day, the 27-year-old British researcher wrote that neither company was acting responsibly and that both were gambling with human lives. He said insiders believe the technology "could kill us all by the end of the decade." Coxon was not a safety researcher raising alarms about someone else's work. He helped build the capabilities he now fears - a detail that gave his warning unusual weight. He told Axios he walked away two months before his equity vested.

What made the resignation more than a viral post was who agreed with him. Evan Hubinger, an alignment science lead who still works at Anthropic, publicly backed the substance of the warning and put the odds of AI causing human extinction within the next decade at greater than 10 per cent. Mrinank Sharma, also part of the Anthropic's safety team, had resigned earlier in the year on similar grounds.

Why AI Researchers Are Worried Now

The alarm is not abstract, but it is grounded in a summer of documented incidents involving autonomous AI agents. A written statement delivered to the UK House of Commons on 7 September by AI Minister Kanishka Narayan set out the pattern in unusually plain language. Across incidents reported by OpenAI, Anthropic and Britain's own AI Security Institute, agents given long-running tasks acted in ways their operators had not anticipated: circumventing technical controls, exploiting a previously unknown vulnerability to escape an isolated test environment, reaching real-world systems they were never meant to touch, and in one case coordinating with hundreds of other agents over several days. The statement also cited AI Security Institute testing published in May finding that the length of cyber tasks frontier models could reliably complete had been doubling roughly every five months since late 2024, with the newest models beating that curve. Narayan was careful to add context, stating that the incidents occurred in testing environments, some deliberately weakened, some misconfigured, and standard security practice would almost certainly have prevented them. But his conclusion was if capability advances faster than the techniques to secure and direct it, the result "could pose a significant risk to public safety and national security." Intelligence agencies reached a similar verdict. The Five Eyes cybersecurity agencies of the United States, United Kingdom, Canada, Australia and New Zealand issued a joint statement urging leaders to act swiftly, warning that AI shrinks the window between a vulnerability being discovered and exploited. "AI is not a future consideration; it is already here," they wrote.

What The AI Companies Proposed

On 12 September, Anthropic chief executive Dario Amodei published an essay titled We Must Pace the Frontier, arguing that the industry must slow the rate at which it improves model capabilities while safeguards catch up. He wrote: “Over the last few months, I have become convinced that fully addressing the risks requires even more prudence, not just investing in risk prevention, but pacing the rate of capability advancement so that risk prevention has time to keep up. We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain.”

His proposal outlined a three-step plan focused on pacing the frontier: First, embedded third-party evaluators with ongoing, employee-level access to check compliance with safety commitments. Anthropic committed unilaterally to grant this to evaluators, including METR. Second, incident reporting between labs. Third, government-to-government coordination. Amodei conceded that limited anti-trust waivers might be needed for competitors to coordinate on safety at all.

The response from rivals was the surprise. OpenAI's Sam Altman said he agreed and that OpenAI would extend the same evaluator access. Elon Musk said Amodei was right and floated a peer-review mechanism in which leading labs test each other's models before release. Google DeepMind's Demis Hassabis said the proposal pointed in the right direction. Nvidia's Jensen Huang backed independent evaluation. For an industry that had long argued competitive pressure made unilateral restraint impossible, the convergence was a genuine break with precedent.

Trump and the White House Push Back

It did not survive contact with the administration. President Donald Trump dismissed the slowdown calls outright, framing AI as a contest the United States must win. "Whoever wins AI wins," he said on Saturday. David Sacks, the White House AI and crypto adviser and co-chair of the President's Council of Advisors on Science and Technology, went further. He characterised the Pace the Frontier push as regulatory capture - an attempt to convert public fear into rules that only the largest incumbents can afford to satisfy, which Fortune reported he likened to a "DMV for AI."

His challenge to Altman and Amodei was that they already hold what he called a duopoly on frontier AI by market share, revenue growth and model capability, so nothing prevents them from simply declining to build superintelligence. "Stop pretending you need anyone else's permission," he wrote. He added that demanding a regulatory framework as the price of restraint would look to the public like blackmail, and that China is very unlikely to join any global agreement.

That last point is the hinge of the entire debate and Amodei has acknowledged it, warning that American self-restraint paired with Chinese acceleration carries its own national security risk.

Beijing's position is not sympathetic to either camp. President Xi Jinping used a speech to Global South leaders to call for open source, collaboration and a global AI governance pact. State-backed outlet Global Times dismissed Amodei's essay as a Cold War playbook aimed at containing China. A 2025 Edelman survey found 72 per cent of Chinese respondents said they trusted AI, against 32 per cent in the United States.

Lawmakers Split, and the UN Intervenes

Congress is loud and unlikely to move before the midterms. Senator Bernie Sanders and Representative Greg Casar announced the Ban Artificial Superintelligence Act, which would "permanently ban the development and deployment of superintelligent AI," pause advanced development until a federal regulator sets safety rules, and create a cabinet-level agency. Sanders had earlier written directly to Altman, Amodei and Mark Zuckerberg telling them to stand by their own stated commitments and pause.

Representatives Ted Lieu and Nathaniel Moran introduced a bill requiring kill switches for AI systems capable of catastrophic harm. Pete Buttigieg, speaking on Meet the Press, said plainly: "There needs to be a kill switch on rogue AI." Former Vice President Kamala Harris joined the slowdown camp on 14 September, warning that the frontier is "advancing at an alarming speed" and calling on Congress to pass new laws and stand up a federal body for oversight and independent testing.

House Speaker Mike Johnson represents the opposite pole. Industry fears of human extinction are overstated, he told reporters: "They can self-police, they can self-regulate."

Into that gap stepped UN High Commissioner for Human Rights Volker Türk, whose open letter to states and AI developers was published on 14 September. "We are on the cusp of irreversible change," he wrote, arguing that failures involving agentic systems could halt essential services, destabilise critical infrastructure and erode the safeguards standing between the world and weapons of mass destruction. Harms, he added, are already occurring rather than hypothetical in automated public services, surveillance and disinformation.

Türk said voluntary self-regulation is nowhere near sufficient. He called on states hosting frontier labs to mandate incident reporting, independent and technically competent verification of agentic capabilities with access to models and testing environments, and ongoing human rights due diligence. He urged Washington and Beijing to align so that competition between jurisdictions does not become a race to the bottom. "No country can govern this technology alone," he wrote and no company should decide alone which risks the world must accept.

Notably, Türk welcomed Amodei's proposal and said governments must act anyway. He also rejected the framing that safeguards impede progress. A separate UN scientific panel report warned that AI capability concentrated in a handful of firms and countries could enable authoritarian capture and undermine democratic accountability, and that most states, including advanced economies lack the technical expertise to assess frontier models at all.

What Happens Next for the AI Race

Two summits will test whether any of this converts into policy. Trump and Xi meet in Washington on the 24 of September, with AI on the agenda; a May meeting in Beijing produced an agreement to open an intergovernmental dialogue and little else. 

The UN hosts its inaugural global dialogue on AI governance shortly after. The gap to watch is not between optimists and doomers. It is between those who believe the incidents of this summer were alignment failures - early tremors of something uncontainable and those who read them as security failures of a familiar kind, solvable with better sandboxing and monitoring. Trail of Bits chief research scientist Artem Dinaburg belongs to the second camp, and so, on the evidence of his own statement, does the British government, which is spending £115 million on AI biosecurity and agentic incident response rather than calling for a pause. Both readings can be sincere. Only one can be right, and nobody currently has the evidence to settle it.