Staff Spotlight: Dr Sepeedeh Shahbeigi

Can you tell us about your research interests?

I am a researcher in AI safety assurance. My work focuses on ensuring that machine learning components in safety-critical systems, such as self-driving cars, operate reliably and safely. My academic journey began when I worked in sensor design and applied for a PhD which switched my focus from hardware to software in autonomous driving system applications and explored methods to ensure that the inputs from the perception sensors to the rest of the system are safe. After my PhD, I joined the AAIP working on developing an autonomous drone use case for a dynamic assurance safety case before moving into my current RA role. 

Currently, my research focuses on deriving safety requirements for AI components in autonomous systems. I aim to define what good safety requirements for AI components look like and develop a systematic approach to specify them, ensuring AI-based systems operate safely.

Why is focusing on specifying safety requirements for AI components important?

AI is increasingly used in safety-critical applications, such as autonomous driving, medical diagnosis or shipping. However, unlike traditional software, ML does not follow fixed, rule-based logic, instead, it learns patterns from data which can make its behaviour unpredictable in new or unseen situations. This unpredictability makes it challenging to assure safety, which is why clear and well-defined safety requirements are essential. Safety requirements set expectations for how an AI system should behave under different conditions.

For example, imagine a self-driving car approaching a pedestrian crossing. If its AI component fails to detect them because daytime conditions were not considered in its development, the consequences could be catastrophic. By defining comprehensive and correct safety requirements on AI components that ensure the self-driving car accurately detects pedestrians in all lighting and weather conditions, we can reduce risks.

Why is defining the requirements for AI challenging?

AI requirements can be challenging because even the foundation - such as what we think constitutes useful data - is quite unclear. Using the guidance set out in AMLAS, we can see that there are four criteria proposed for a good dataset: completeness, relevance, accuracy, balance. But what are the metrics and how to measure these criteria in a way that is relevant to safety, is an open question. A dataset that’s 99% balanced might be sufficient for one task, but inadequate for another. Without a clear definition, we often rely on marginal accuracy improvements without truly addressing whether the dataset meets the system’s needs. Even if a dataset is “good” at a given time, there's no guarantee it will remain valid as the world evolves - what we define as good performance today might not hold true tomorrow.

Beyond data quality, there’s the issue of defining measurable and meaningful requirements in unpredictable real world environments. Saying something like “the system should never hit an object” sounds good in theory, but what does that mean operationally? The world is full of edge cases, like fog impairing visibility or unusual objects such as vehicles with moving billboards that mimic human motion, where the system may register objects where none exist. A model might work fine in a limited simulation, but fail in an open world scenario with unpredictable inputs. We lack methods for systematically evaluating whether we're considering all important edge cases - or how to measure if we’re missing something crucial. 

My research into this topic so far has been focused on the challenge of specifying safety requirements for ML components within complex, real-world systems, while ensuring their traceability. Traditional safety engineering approaches often struggle when applied to ML due to its data-driven nature, and this has influenced my work exploring how safety requirements can be understood, specified, and validated in ways that align with the characteristics of AI development. This is a challenge that is common to many ML applications, so my work will be relevant well beyond my specific projects.

What are the biggest changes you’ve witnessed since working in the field of autonomous driving?

The field has undergone a noticeable shift from focusing on isolated technical achievements to a broader, more safety-aware perspective. There was a lot of enthusiasm around improving specific capabilities like detection accuracy, but these improvements were often carried out in isolation and often lacked a clear connection to the overall system. Safety considerations were rarely prioritised, with most of the attention placed on pushing performance metrics.

Over the past few years, I’ve seen a significant increase in awareness about the hazards autonomous systems can pose. This has been partly driven by high-profile incidents, which highlighted the consequences of not properly assessing safety risks. People now realise that it’s not enough for a model to perform well in typical scenarios - it must also be able to handle rare or never-before-seen situations. Discussions around runtime monitoring, for example, are gaining traction as one way to address the unpredictability of ML in open-world environments.

In my work, I focus on the design stage of the ML model to make sure these components don’t contribute to system-level hazards. We've begun to see developments like models being able to say “I don’t know” when faced with unfamiliar inputs, showing an important shift from the older mindset where models were forced to choose between a hard yes or no, regardless of confidence. This is a positive sign that the field is beginning to prioritise uncertainty and context in safety-critical decisions. Challenges remain, particularly in understanding how small failure rates at the ML level might translate to significant system level risks - like misclassifying a sign in a crowded urban setting versus on an empty highway - but it's been encouraging to see the changes to the industry even as recently as the last few months.

Why is it important, when thinking about safety assurance, that safety is inclusive for all?

One of the key areas we have to think about when creating safety assurance cases is identifying a wide range of risks and scenarios. It’s not enough to focus on only the most obvious hazards - such as the misidentification of objects in the case of autonomous driving - we have to consider less predictable risks as well. This is why it is important to ensure that safety is inclusive to all as, when it is, it allows us to consider a wealth of different perspectives and experiences that help inform and shape our work.

For example, making sure that the data any ML system uses to make decisions is diverse and representative and working to remove any data that could cause an ML system to show bias, or make incorrect decisions, i.e. in healthcare settings.  If we build systems without considering a diverse range of perspectives, whole areas of the human experience are missed, leading to limited safety requirements that could be hazardous to wide sections of society.

Finally, where can we find you when you’re not working?

When I’m not working, I like staying active and trying new things. I enjoy hiking, whether it’s an easy trail like walking around York or something more challenging with great views. I also love dining out and exploring new restaurants, especially places with interesting flavours. This often inspires me to experiment with different cuisines at home. Recently, I joined an allotment and have been getting into gardening, which has been both relaxing and rewarding. I’m especially excited about the strawberries I’ve planted—if all goes well, I’ll be bringing plenty into the office this summer!