UK must change approach to safety to meet PM's AI goals

Posted on Thursday 24 September 2026

Addressing the UN General Assembly this week, Prime Minister Andy Burnham calls for a “single set of global principles and standards” on AI. Only a day later, it was revealed that an Open AI agent hacked a government healthcare portal in Australia. 

The growing divide between the technological advancements of AI and global efforts around regulations and standards is something the CfAA has been involved in through its work in domains such as maritime and automotive autonomy. 

Burnham’s speech sets out ambitious plans for the UK to lead international efforts on AI standards. On this the UK is already well placed, the question is less ‘can we’ and more how do we leverage existing work in this area and improve on what the UK is already doing well. 

For example, the UK has multiple institutes working on safety of AI and autonomy, the CfAA alone has been working on safety assurance of these systems since 2018. However safety has, all too often, taken a backseat. Viewed as stifling innovation and slowing growth. 

Security breaches, though, are on the increase. In the Hugging Face case the models were being tested to identify their ability to launch security attacks (and not properly isolated from the internet); similarly, in the new Australian case the models were instructed to ‘trawl’ the internet looking for data on Australian drug usage. The models went beyond what their developers intended in their actions, but they didn’t “decide for themselves” what to do.  In both cases, the incidents occurred during OpenAI’s internal evaluation of its models. The models persistently attempted to look up the answers to the questions or tasks they were given, which ultimately led to the attacks. In the Hugging Face case, the model believed that the answer was stored on Hugging Face; in the Australian case, the model attempted to access the “answer” after encountering “repeated blocks” in its attempts regarding questions about public medicine spending in Australia. 

Whilst safety is paramount, it’s also extremely important to have a clear and balanced explanation of these issues so the public is better informed and policy-makers have an accurate understanding of the issues.

We cannot continue to view safety as a ‘late-stage addition’ if we are to achieve the PM’s goals of leading in this space. We know that safety assurance cannot be a “stand off” activity. In the case of Frontier models or Agentic AI it has to be achieved by actively engaging and shaping the way the agentic systems work. We also understand the complexities around shaping and agreeing global standards in complex industries, something highlighted by our recent work in the Singapore maritime space.

We fully support the PM’s position, both on the seriousness of the risks posed and the need for the UK to take a leading position on safety and security of AI.

To achieve this we recommend: 

1. A working group of the UK’s safe AI research centres working directly with the new  Office for AI to share latest safety research and evidence from investigating incidents and support policy-making decisions

2. Greater dedicated funding for specific research related to Frontier and Agentic AI models including their potential impact on physical safety of the public and impact on critical national infrastructure.

3. Clear communication protocols that balances the sensationalisation in the press and ensures the public and policy-makers have clear insight and transparency around any AI-related incidents.