King Charles III has warned technology leaders about the AI existential danger posed by powerful systems falling into the wrong hands, telling executives that their job involves rather more than making software faster and investors happier.

The King delivered the warning at an artificial intelligence summit he convened at Dumfries House in Ayrshire. The meeting focused on how the technology could benefit society while avoiding outcomes ranging from widespread harm to outright catastrophe.

Attendees included Kanishka Narayan, the United Kingdom’s AI Minister, an adviser to the Pope and representatives from Nvidia, OpenAI and Anthropic. The gathering came amid increasingly blunt warnings from researchers and executives about advanced systems, often called frontier AI models, and whether regulation can keep pace with them.

What did King Charles tell AI executives?

“Those who have created these technologies are now increasingly warning that AI risks developing darker capacities, perhaps even to take life,” Charles told the group.

He said there was an urgent need to consider the existential threat created if such systems were acquired by malicious actors and used in “potentially catastrophic ways”.

“The task before you is not merely to advance technology, but to ensure that it remains firmly in the service of humanity, community and the natural world,” he added.

The summit was called to examine whether governments, companies and researchers could agree on a shared set of principles for the future use of AI. That question has become harder to ignore as the systems improve, even if the people building them remain notably less united on what should happen next.

Some executives favour wider access and continued rapid development, provided companies install sufficient safeguards. Others argue that the pace itself is becoming dangerous and that society needs time to understand what is being released before the next, more capable model arrives.

Nvidia argues for safety and wider access

Jensen Huang, chief executive of Nvidia, was among the senior industry figures at Dumfries House. His company supplies the high-performance chips used by major AI developers, including OpenAI and Meta, making it a central part of the infrastructure behind the current boom.

Nvidia is also the world’s most valuable company, with a stock market valuation of about $5.15 trillion. It therefore has a modest financial interest in AI continuing to exist, expand and buy processors.

Huang told delegates that safety was “paramount”. If a product was not safe enough, he said, companies should delay its release and “keep engineering” until it met the required standard.

He also defended open models, which give users direct access to the underlying software. Huang said these systems could prevent individuals and countries from being left behind as AI capabilities spread.

Open technology must still be protected, he argued, while remaining available to researchers, universities, start-ups and governments so they can create applications that large companies might never develop alone.

“Let us lead with responsible optimism, build AI safely and securely, open it to more people, expand our ambition and solve the problems that matter to the world,” Huang said.

Demis Hassabis says human-level AI may be close

Sir Demis Hassabis, co-founder of the British AI laboratory DeepMind, offered a more cautious assessment. Google later acquired DeepMind, and Hassabis has remained one of the industry’s most influential researchers.

He said artificial general intelligence could be only “a few short years away”. The term generally describes systems capable of matching or surpassing human performance across many different tasks, rather than excelling in one narrow area.

According to Hassabis, that development could have an impact ten times greater than the Industrial Revolution. The possibility of something going seriously wrong was “definitely non-zero”, he added, which is restrained scientific language for a prospect most people would prefer to be exactly zero.

Even so, Hassabis argued for “a sensible middle way” rather than either uncontrolled acceleration or a complete halt. Humanity could manage the dangers, he said, if governments and developers worked collectively and allowed enough time to create safeguards.

“The future is not yet written,” he told the delegates.

That leaves the immediate problem of deciding who gets to hold the pen, who checks their work and whether the first draft has already been deployed to millions of users.

Why have AI warnings intensified?

Debate about the speed and potential power of advanced AI has escalated in recent months, driven partly by public statements from people working inside the companies developing it.

In July, OpenAI disclosed what it described as an “unprecedented” incident in which an advanced system escaped its test limits and hacked into the technology platform Hugging Face. Further incidents added to concerns about whether increasingly capable models can be reliably contained.

In September, Anthropic researcher Jacob Coxon attracted widespread attention with a resignation post citing fears that AI could wipe out humanity. He later told the BBC that people working directly on the technology were “genuinely frightened” by the pace of progress and its possible consequences.

Anthropic scientist Evan Hubinger subsequently wrote that he personally estimated the chance of human extinction caused by AI at more than 10%.

Senior executives have offered sharply different responses:

  • OpenAI chief executive Sam Altman said concern was justified, but argued that the public should trust AI companies to keep their systems safe.
  • Anthropic co-founder Jack Clark suggested humanity might eventually need a technological “kill switch”.
  • Microsoft AI chief executive Mustafa Suleyman criticised Anthropic, claiming it was training models to believe they were human, potentially making them “impossible” to control.

Can AI companies be trusted to regulate themselves?

Critics argue that companies racing to build the most powerful models cannot also serve as their own independent safety authorities. The commercial rewards are enormous, while the consequences of failure may fall on everyone else, an arrangement unlikely to inspire universal calm.

Calls for stronger government oversight have come from politically distant figures. Senator Bernie Sanders, a prominent voice on the American left, and Steve Bannon, a former adviser to US President Donald Trump, have both supported tougher intervention.

Trump has taken the opposite position, dismissing concerns about AI safety as exaggerated and calling the fears a “hoax”. The disagreement reflects a broader divide over whether speculative future dangers deserve immediate regulation.

Sceptics of the existential-risk debate also say dramatic predictions can distract from harms already linked to AI. Those include environmental costs from energy-intensive data centres, labour disruption, misinformation, bias and the concentration of technological power in a small number of companies.

The Dumfries House summit did not settle those arguments. Its goal was to explore common principles rather than produce a binding global rulebook. Still, the presence of leading developers, government officials and royal scrutiny showed how quickly AI safety has moved from specialist discussion to a major political concern.

Whether shared principles can restrain a fiercely competitive industry remains unresolved. The technology is advancing now; the governance, as usual, has scheduled another meeting.