Architectural approach needed for safe Agentic AI

Posted on Tuesday 25 August 2026

The popularity of agentic AI continues, leading to increased and critical questions about the safety of such technologies. Whilst these are predominantly digital-only, the possibilities of their use in cyber-physical systems (CPS) make them an area of growing interest to the CfAA. 

Understanding agentic AI

Agents are autonomous in that they can operate with little or no human input or control and, significantly, they can take actions that otherwise might be undertaken by humans, e.g. make payments. The mainstream in agentic AI is using Large Language Models (LLMs) as the “brain” of the agent; planning, invoking external tools, such as web searches or an on-line bank’s API, and collaborating with other agents. 

Until recently, agents could only undertake tasks of a few seconds duration. At present agents can undertake tasks running for tens of minutes - with this doubling every 7 months. The speed of this advancement is likely to have an impact on the difference in lab and real world accuracy - this matters because if accuracy is not rising - or even degrading - how can we assure the adequacy (including safety) of the behaviours especially as there is evidence that agents are more likely to diverge from their objectives the longer they run? This is the key question which is of interest to the CfAA and our work on safety assurance. 

New challenges continue to emerge

We don't yet know the extent of the challenges agentic AI may bring, but two of the biggest that industry and regulators are facing are around assurance and dynamism. For example, there is limited ability to directly assure the agents themselves (the LLMs at their core are too complex) so the approach needs to be architectural, putting in place guardrails—in agentic AI the term “control plane” is sometimes used—to ensure that the collection of agents behaves as desired; this in turn is hard due to the growing scope of agent “authority”, for example, ability to create new agents, hence the range of activities that need to be monitored and controlled. 

The dynamism of such agents makes it harder to identify and manage changes. For example, agent behaviours can change over time as they “learn” how to do the job better, and the environment can change, e.g. a tool used by the agent is updated to have different capabilities, which might mean the impact of the agent is different. 

Exploring approaches to safety 

Safety analysis is fundamentally concerned with modelling cause and effect - regardless of the underlying technology or application domain. This is why one approach is to consider Structural Causal World Models and their potential to model the complex causal relationships and interdependencies in an agentic AI system. SCWMs could enable their safety to be assessed and predicted and, importantly, offer the ability to specify controls to reduce the likelihood of undesirable outcomes from use of the agents. In effect, playing an important role in defining and validating that control plane mentioned above. SCWM can provide the link between the agentic AI behaviour in the computer and its physical consequences - making them particularly useful if and when agentic AI is used in CPS. 

Whilst the area of agentic AI is 'new' its clear that many of the principles of safety remain. Namely, that safety assurance cannot be a “stand off” activity, it has to be achieved by actively engaging and shaping the way the agentic systems work. It's this which will form the CfAA's work in this area, both integrating agentic AI into our research programmes and getting involved in the building of such systems, bringing us many interesting avenues to explore.