DC3 -- A Diagnostic Case Challenge Collection for Clinical Decision Support

TL;DR

DC3 provides a curated collection of 31 complex diagnostic cases with expert solutions and relevance annotations, aiming to improve evaluation and development of clinical decision support systems.

Contribution

This paper introduces DC3, a novel benchmark dataset of difficult diagnostic cases with expert annotations and biomedical literature relevance, addressing evaluation gaps in clinical decision support.

Findings

Contains 31 challenging diagnostic cases with expert solutions

Includes relevance judgments linking cases to biomedical literature

Facilitates standardized evaluation of decision support systems

Abstract

In clinical care, obtaining a correct diagnosis is the first step towards successful treatment and, ultimately, recovery. Depending on the complexity of the case, the diagnostic phase can be lengthy and ridden with errors and delays. Such errors have a high likelihood to cause patients severe harm or even lead to their death and are estimated to cost the U.S. healthcare system several hundred billion dollars each year. To avoid diagnostic errors, physicians increasingly rely on diagnostic decision support systems drawing from heuristics, historic cases, textbooks, clinical guidelines and scholarly biomedical literature. The evaluation of such systems, however, is often conducted in an ad-hoc fashion, using non-transparent methodology, and proprietary data. This paper presents DC3, a collection of 31 extremely difficult diagnostic case challenges, manually compiled and solved by clinical experts. For each case, we present a number of temporally ordered physician-generated observations alongside the eventually confirmed true diagnosis. We additionally provide inferred dense relevance judgments for these cases among the PubMed collection of 27 million scholarly biomedical articles.