1.3.1 Validation of safety requirements

Author: SAM demonstrator project

The introduction of artificial intelligence (AI) and machine learning (ML) applications into clinical systems can create challenges for traditional design approaches that require clearly defined and precise specifications of the operating environment, operational scenarios and of the resulting safety requirements that bound the behaviour of the AI / ML system. Healthcare is a complex domain, and clinical systems are made up of many different actors and technologies all interacting with one another in ways that can be very dynamic and responsive to the requirements of a specific situation.

One way of both eliciting and validating safety requirements (though not the only one, nor even a standalone approach) is to seek input and feedback from stakeholders (e.g. through simulation or interviews). This approach is particularly appropriate where human-machine interaction and training are concerned. 

For example, in the case of the design of an autonomous infusion pump to be used in intensive care, input from stakeholders might produce training requirements as shown below. Feedback from stakeholders on a design prototype can then provide information about the extent to which the requirements have been met. 

Similarly, high-level safety requirements relating to autonomy and control might be validated through feedback on a proposed interaction design.