Authors:
Derek Sleeman
1
;
Kiril Kostadinov
1
;
Laura Moss
2
;
1
and
Malcolm Sim
2
Affiliations:
1
Computing Science Department, The University of Aberdeen, AB24 3FX, U.K.
;
2
Academic Unit of Anaesthesia, Pain and Critical Care, School of Medicine, University of Glasgow, G31 2ER, U.K.
Keyword(s):
Expertize Capture, Clinical Decision Support Systems, Inconsistencies, Negotiations, DELPHI-approach, Refinement, Cognitive Informatics, Medical Informatics, Pattern Recognition and Machine Learning.
Abstract:
Knowledge intensive clinical systems, as well as machine learning algorithms, have become more widely used over the last decade or so. These systems often need access to sizable labelled datasets which could be more useful if their instances are accurately labelled / annotated. A variety of approaches, including statistical ones, have been used to label instances. In this paper, we discuss the use of domain experts, in this case clinicians, to perform this task. Here we recognize that even highly rated domain experts can have differences of opinion on certain instances; we discuss a system inspired by the Delphi approaches which helps experts resolve their differences of opinion on classification tasks. The focus of this paper is the IS-DELPHI tool which we have implemented to address the labelling issue; we report its use in a medical domain in a study involving 12 Intensive Care Unit clinicians. The several pairs of experts initially disagreed on the classification of 11 instances
but as a result of using IS-DELPHI all those disagreements were resolved. From participant feedback (questionnaires), we have concluded that the medical experts understood the task and were comfortable with the functionality provided by IS-DELPHI. We plan to further enhance the system’s capabilities and usability, and then use IS-DELPHI, which is a domain independent tool, in a number of further medical domains.
(More)