1.2.1 Considering human-machine interaction

Author: Dr Mark Sujan

AI is seen as the next IT step in addressing health and care challenges. If done well, AI could support clinicians in their decision-making, it could generate predictions aimed at improving inefficiencies in the management of care processes, and it might radically transform the way care is provided and accessed (Topol, 2019). Diagnostic devices using AI are leading the way (e.g. AI applications interpreting radiological images to identify diabetic retinopathy, Abràmoff et al. 2018 or to differentiate COVID-19 from other types of chest infections, Li et al. 2020). Other examples of healthcare AI applications include the use of patient-facing chatbots, mental health applications, ambulance service triage, sepsis diagnosis and prognosis, patient scheduling, planning of resources, quality improvement activities, and even the development of COVID-19 vaccines.

However, the aspiration of using AI to improve the efficiency of health systems and to enhance patient safety, patient experience and staff wellbeing is currently weakened by a narrow focus on technology that contrasts people and AI (“human vs. machine”) and by a limited evidence base of AI in real-world use (Nagendran et al., 2020).

It is likely that the real challenges for the adoption of AI will arise when algorithms are integrated into healthcare systems to deliver a service in collaboration with healthcare professionals as well as other technology. It is at this health system level, where teams consisting of healthcare professionals and AI systems cooperate and collaborate to provide a service, that Human Factors and Ergonomics (HF/E) challenges will come to the fore (Sujan et al., 2019). This leaves gaps in the assurance of safety of AI, which should be addressed using rigorous HF/E approaches.

HF/E is a scientific discipline that is concerned with the design of sociotechnical systems to improve overall system performance, safety and the wellbeing of people. From this perspective, AI is regarded as one element of the sociotechnical system. HF/E provides theories and methods to support the design and use of AI during its lifecycle as part of the wider system.

Critical HF/E considerations for the successful use of AI in healthcare include (Sujan et al., 2021):

1. Situation awareness: design options need to consider how AI can support, rather than erode, people’s situation awareness.
2. Workload: the impact of AI on workload needs to be assessed because AI can both reduce as well as increase workload in certain situations.
3. Automation bias: strategies need to be considered to guard against people relying uncritically on the AI (e.g. the use of explanation and training).
4. Explanation and trust: AI applications should explain their behaviour and allow users to query it in order to reduce automation bias and to support trust.
5. Human-AI teaming: AI applications should be capable of good teamworking behaviours to support shared mental models and situation awareness.
6. Training: people require opportunities to practise and retain their skill sets when AI is introduced, and they need to have a baseline understanding of how the AI works.
7. Relationships between staff and patients: the impact on relationships needs to be considered (e.g. whether staff will be working away from the patient once more and more AI is introduced).
8. Ethics: AI needs to be robust and ethical with a focus on human autonomy, reduction of bias, safety, privacy and transparency.

Designers and developers of AI, individuals with responsibility for procuring AI applications, regulators, and bodies funding research and development need to move beyond the technology-centric view, and instead approach AI from a systems perspective (i.e. to consider from the outset the interaction of people with AI as part of the wider clinical and health system).

These activities need to be underpinned by education in and support with HF/E, which healthcare professionals and organisations can tap into.

For further details see: Sujan M, Baber C, Salmon P, Pool R, Chozos N. Human Factors and Ergonomics in Healthcare AI. Wootton Waven: Chartered Institute of Ergonomics and Human Factors; 2021

References

• ABRÀMOFF, M. D., LAVIN, P. T., BIRCH, M., SHAH, N. & FOLK, J. C. 2018. Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. npj Digital Medicine, 1, 39.
• LI, L., QIN, L., XU, Z., YIN, Y., WANG, X., KONG, B., BAI, J., LU, Y., FANG, Z., SONG, Q., CAO, K., LIU, D., WANG, G., XU, Q., FANG, X., ZHANG, S., XIA, J. & XIA, J. 2020. Artificial Intelligence Distinguishes COVID-19 from Community Acquired Pneumonia on Chest CT. Radiology, 200905.
• NAGENDRAN, M., CHEN, Y., LOVEJOY, C. A., GORDON, A. C., KOMOROWSKI, M., HARVEY, H., TOPOL, E. J., IOANNIDIS, J. P. A., COLLINS, G. S. & MARUTHAPPU, M. 2020. Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies. BMJ, 368, m689.
• SUJAN, M., BABER, C., SALMON, P., POOL, R. & CHOZOS, N. 2021. Human Factors and Ergonomics in Healthcare AI. Wootton Waven: Chartered Institute of Ergonomics and Human Factors.
• SUJAN, M., FURNISS, D., GRUNDY, K., GRUNDY, H., NELSON, D., ELLIOTT, M., WHITE, S., HABLI, I. & REYNOLDS, N. 2019. Human factors challenges for the safe use of artificial intelligence in patient care. BMJ Health & Care Informatics, 26, e100081.
• TOPOL, E. J. 2019. High-performance medicine: the convergence of human and artificial intelligence. Nature Medicine, 25, 44-56.