World of ScoreCraft: Novel Multi‐Scorer Experiment on the Impact of a Decision Support System in Sleep Staging

TL;DR

This study explores how decision support systems affect sleep scoring accuracy and finds that accurate recommendations improve results regardless of their source.

Contribution

A novel online platform was used to investigate the impact of recommendation sources on sleep staging accuracy and decision-making.

Findings

Correct recommendations significantly improved scoring accuracy for both traditional and self-applied PSGs.

No significant bias was observed toward AI-generated versus human-generated recommendations.

Traditional PSGs were scored slightly more accurately than self-applied PSGs, though the difference was not statistically significant.

Abstract

Manual scoring of polysomnography (PSG) is a time‐intensive task, prone to inter‐scorer variability that can impact diagnostic reliability. This study investigates the integration of decision support systems (DSS) into PSG scoring workflows, focusing on their effects on accuracy, scoring time and potential biases toward recommendations from artificial intelligence (AI) compared to human‐generated recommendations. Using a novel online scoring platform, we conducted a repeated‐measures study with sleep technologists, who scored traditional and self‐applied PSGs. Participants were occasionally presented with recommendations labelled as either human‐ or AI‐generated. As the goal of this study was to isolate the effect of perceived recommendation sources on scorer behaviour, all recommendations were human‐generated. We found that traditional PSGs tended to be scored slightly more accurately than self‐applied PSGs, but this difference was not statistically significant. Correct recommendations significantly improved scoring accuracy for both PSG types, while incorrect recommendations reduced accuracy. No significant bias was observed toward or against AI‐generated recommendations compared to human‐generated recommendations. These findings highlight the potential of DSSs to enhance PSG scoring reliability. However, ensuring the accuracy of the suggestions is critical to maximising its benefits. Future research should explore the long‐term impacts of DSS on scoring workflows and strategies for integrating AI in clinical practice.

Figures