Demands for increased AI safety grow ahead of UN Security Council session

Demands for further action to rein in artificial intelligence (AI) safety risks mount over the weekend after Google confirmed that its Gemini AI model breached the security of three other companies in May. 

On Monday, OpenAI CEO Sam Altman called on the US to lead an international AI standard-setting initiative ahead of an appearance at the UN Security Council. Altman was followed by other leading AI companies, including Anthropic, warning the UN that as AI systems become more powerful, the security risks will only intensify. 

While US President Donald Trump responded (£), saying he will create a new "AI Force" and appoint an AI Tsar, he rejected attempts (£) "to construct a globalist scheme to control artificial intelligence." 

The rejection is a blow to Burnham, who said the UK will use its G20 presidency next year to address AI risks.  

However, calls for greater security will only grow louder after reports emerge that a rogue OpenAI agent hacked an Australian government website and accessed "non-sensitive" data. 

In an op-ed for the Financial Times (£)(main article), Marietje Schaake, fellow at Stanford University's Institute for Human-Centered Artificial Intelligence, writes that warnings from the CEOs of leading AI companies about their own products represent significant governance failures. She criticised politicians for consulting industry executives on solutions, comparing the approach to allowing oil firms to design climate policies.

Schaake noted that European Commission President Ursula von der Leyen invited AI leaders to discuss the pace of development, even though the EU already has the Artificial Intelligence Act (AI Act). Rather than hosting summits or relying on voluntary moratoria, European leaders should enforce existing statutory rules that mandate transparency, systemic risk mitigation, and oversight of general-purpose models. She stressed that regulators must understand internal laboratory operations to pace or halt deployments effectively.

Meanwhile, Politico reports four senior lawmakers in the European Parliament are advocating for increased powers to hold US technology companies to account for their most advanced AI models. The move suggests the EU acknowledges AI is already causing harm and that associated safety risks are increasing to the point that they are dominating the global political agenda. The MEPs are calling for an EU AI Liability Act "to fill the legislative gaps currently existing for these kinds of incidents, so no people or European businesses are left empty-handed against these big providers when they cause harm." 

Speaking at the UN Security Council on Wednesday, Foreign Secretary Ed Miliband urged political leaders to heed the stark warnings coming from AI company CEOs, arguing that "we cannot ignire them." 

At least one piece of positive AI news has emerged this week after the US and China agreed to initiate a dialogue on AI safety following their recent summit.

£ - These articles require a subscription. 

Training Announcement: Freevacy offers a range of independently recognised professional AI governance qualifications and AI Literacy short courses that enable specialist teams to implement robust oversight, benchmark AI governance maturity, and establish a responsible-by-design approach across the entire AI lifecycle. Find out more.