A machine learning system to optimise triage in an adult ophthalmic emergency department: a model development and validation study

Background: A substantial proportion of attendances to ophthalmic emergency departments are for non-urgent presentations. We developed and evaluated a machine learning system (DemDx Ophthalmology Triage System: DOTS) to optimise triage, with the aim of reducing inappropriate emergency attendances and streamlining case referral when necessary.

Methods: DOTS was built using retrospective tabular data from 11,315 attendances between July 1st, 2021, to June 15th, 2022 at Moorfields Eye Hospital Emergency Department (MEH) in London, UK. Demographic and clinical features were used as inputs and a triage recommendation was given ("see immediately", "see within a week", or "see electively"). DOTS was validated temporally and compared with triage nurses' performance (1269 attendances at MEH) and validated externally (761 attendances at the Federal University of Minas Gerais - UFMG, Brazil). It was also tested for biases and robustness to variations in disease incidences. All attendances from patients aged at least 18 years with at least one confirmed diagnosis were included in the study.

Findings: For identifying ophthalmic emergency attendances, on temporal validation, DOTS had a sensitivity of 94.5% [95% CI 92.3-96.1] and a specificity of 42.4% [38.8-46.1]. For comparison within the same dataset, triage nurses had a sensitivity of 96.4% [94.5-97.7] and a specificity of 25.1% [22.0-28.5]. On external validation at UFMG, DOTS had a sensitivity of 95.2% [92.5-97.0] and a specificity of 32.2% [27.4-37.0]. In simulated scenarios with varying disease incidences, the sensitivity was ≥92.2% and the specificity was ≥36.8%. No differences in sensitivity were found in subgroups of index of multiple deprivation, but the specificity was higher for Q2 when compared to Q4 (Q4 is less deprived than Q2).

Interpretation: At MEH, DOTS had similar sensitivity to triage nurses in determining attendance priority; however, with a specificity of 17.3% higher, DOTS resulted in lower rates of patients triaged to be seen immediately at emergency. DOTS showed consistent performance in temporal and external validation, in social-demographic subgroups and was robust to varying relative disease incidences. Further trials are necessary to validate these findings. This system will be prospectively evaluated, considering human-computer interaction, in a clinical trial.

Funding: The Artificial Intelligence in Health and Care Award (AI_AWARD01671) of the NHS AI Lab under National Institute for Health and Care Research (NIHR) and the Accelerated Access Collaborative (AAC).