Capturing Requirements for a Data Annotation Tool for Intensive Care: Experimental User-Centered Design Study

TL;DR

This study investigates clinicians' data annotation processes in ICUs to identify key requirements for developing an effective, user-centered digital annotation tool tailored for complex clinical data environments.

Contribution

It provides a detailed analysis of clinicians' annotation workflows and establishes 11 specific requirements for designing a digital annotation tool in healthcare settings.

Findings

Clinicians' annotation methods vary between individuals and cases.

No significant differences in annotation approaches across staff roles.

Identified 11 key requirements for an effective clinical data annotation tool.

Abstract

Intensive care units (ICUs) are complex and data-rich environments. Data routinely collected in the ICUs provides tremendous opportunities for machine learning, but their use comes with significant challenges. Complex problems may require additional input from humans which can be provided through a process of data annotation. Annotation is a complex, time-consuming process that requires domain expertise and technical proficiency. Existing data annotation tools fail to provide an effective solution to this problem. In this study, we investigated clinicians' approach to the annotation task. We focused on establishing the characteristics of the annotation process in the context of clinical data and identifying differences in the annotation workflow between different staff roles. The overall goal was to elicit requirements for a software tool that could facilitate an effective and time-efficient data annotation. We conducted an experiment involving clinicians from the ICUs annotating printed sheets of data. The participants were observed during the task and their actions were analysed in the context of Norman's Interaction Cycle to establish the requirements for the digital tool. The annotation process followed a constant loop of annotation and evaluation, during which participants incrementally analysed and annotated the data. No distinguishable differences were identified between how different staff roles annotate data. We observed preferences towards different methods for applying annotation which varied between different participants and admissions. We established 11 requirements for the digital data annotation tool for the healthcare setting. We conducted a manual data annotation activity to establish the requirements for a digital data annotation tool, characterised the clinicians' approach to annotation and elicited 11 key requirements for effective data annotation software.