The next phase of the global debate over powerful artificial intelligence is taking shape as a struggle over regulatory power: Who sets safety thresholds? Who tests frontier models? And what happens when a company or country crosses an agreed line?
A high-level U.N. Security Council debate revealed growing support for independent testing, common evaluation standards, incident reporting and stronger government oversight. But it also exposed the dispute likely to define the negotiations ahead: whether those safeguards should become internationally agreed rules or remain primarily under national control.
Scientists, technology executives and several governments are no longer speaking only in broad terms about the risks of AI. They are proposing institutions and mechanisms intended to manage them.
At Wednesday’s meeting, Yoshua Bengio, co-chair of the U.N.’s Independent International Scientific Panel on AI, offered one of the most detailed regulatory prescriptions: licensing frontier AI systems, requiring liability insurance and subjecting developers to independent safety assessments before training and deployment.
The central question is whether governments can agree on common safeguards without agreeing on who has authority to enforce them — and what happens when a company or country refuses to comply.
“Frontier AI should be licensed like other critical technologies — medicine, aviation, and nuclear energy — to incentivize safe development,” Bengio said. “As in these domains, liability insurance should be required.”
Bengio also called for mandatory reporting of security incidents, greater scientific independence from AI companies and international agreements aimed at making the technology safe by design.
“We must agree collectively that no one — no company and no country — should develop an AI system that could cause large-scale harm,” he said. “Developers must demonstrate to independent experts that a system is safe to train and safe to deploy.”
OpenAI CEO Sam Altman and Anthropic CEO Dario Amodei did not go as far as Bengio’s call for licensing. But both argued that consequential decisions cannot remain solely with the companies building the most powerful systems.
“If AI is to be democratic, the most important decisions cannot be made by labs in San Francisco alone,” Altman said. “They must be shaped through democratic processes and by governments accountable to the people that they serve.”
Amodei proposed international agreements to prevent AI from being used to develop biological weapons, standards for evaluating and verifying advanced model capabilities, common testing for loss-of-control risks and a notification system for serious misuse.
“No leader, no company and no nation can manage this alone,” Amodei said.
The differences become sharper over how far international oversight should extend. Britain advocated global principles and standards, while France explicitly called for international regulation.
“The companies building the models cannot simply be left to their own devices,” British Foreign Secretary Ed Miliband said. Britain plans to put AI at the center of its Group of 20 presidency next year and work toward “a single set of global principles and standards.”
French Foreign Minister Jean-Noël Barrot made the case more explicitly. “Just as with the atom back then, the international community must now regulate AI in order to make the most of its potential and avoid the worst,” he said.
The dispute over international authority extends beyond the Security Council to the U.N.’s emerging AI institutions and Britain’s Group of 20 presidency next year. Those forums may be able to develop common assessments, testing methods and reporting standards without becoming the global regulator Washington opposes.
China supported international cooperation while warning against technological monopolies. Chinese Ambassador Fu Cong said companies, “especially tech giants,” should subject themselves to legal regulation and public oversight.
“AI is a shared asset for all humanity, and should not become rich countries’ and rich people’s game,” Fu said.
The United States drew the clearest boundary around international action. White House science adviser Michael Kratsios said international institutions should help countries develop their own technological and regulatory capacity, not establish rules above the national level.
“International dialogue, in this forum and in others, cannot be allowed to drift toward global governance,” Kratsios said. “This body and others like it should focus on sharing best practices to build domestic capacity, not establishing a global regulatory regime.”