2.6 Handling change during operation

The specific capability that the AI is providing needs to be defined and characterised in the context of supporting the care pathway.

This must consider the interaction between the AI and the HCP both as a user and also a supervisor.

This must consider the authority limits autonomous AI can have.

• Pro-active or re-active: HCP routinely monitors behaviour of AI or responds to an alert or alarm.
• Frequency: sufficient to maintain situational awareness but not so frequent that it compromises efficiency.
• Trends: does the monitoring indicate progression to an unsafe state or imminent point of handover?
• Escalation: is there a need for a monitoring HCP to seek second opinion or high authority before initiating any action?
• Monitoring architecture: this needs to be defined and consideration given to whether the AI simply monitors itself, whether the AI is monitored independently by another element of the care system or whether it’s a combination of both.

• Definition of safe limits: the limit of authority the AI can have before HCP intervention is required needs to be defined. Definition should consider the need for soft limits (i.e. those that can be transgressed but signify an impending handover requirement). Hard limits (i.e. those that must never be exceeded) need to be defined. The protocol needs to consider the degree of authority the AI has; does it have the same as a HCP or is it restricted to a lower level?
• Definition of transfer state: the AI’s behaviour whilst a handover is being determined needs to be specified. Does the AI continue to perform its function that may result in a change in outcome, does it maintain a steady state of output or default to the previous known good (safe) output? The period of time that the transfer state can persist needs to be defined.
• Definition of safe state: the AI’s behaviour once authority has been relinquished needs to be defined. Does the AI revert to an “off” state and dissociate itself from the care pathway or does it continue to function in “hot standby” in order to support a subsequent transfer of authority back from the HCP?
• Definition of re-engagement criteria: the criteria and process for re-engagement of the AI needs to be defined.
• Definition of audit log: an audit log may be needed to support informed and safe handover of authority to an HCP. It will be necessary to identify those clinical variables, environmental conditions and system parameters that influenced the learning. The HCP will need to be able to quickly assimilate the clinical scenario and take effective mitigating action.