MDDial: A Multi-turn Differential Diagnosis Dialogue Dataset with Reliability Evaluation

TL;DR

MDDial is the first publicly available English dataset for differential diagnosis dialogue systems, introducing a unified reliability score and evaluating language models' performance in medical diagnosis dialogues.

Contribution

The paper presents MDDial, a novel English dataset for ADD, and proposes a unified score for diagnosis reliability, addressing gaps in existing datasets and evaluation methods.

Findings

Language models perform well on general NLP tasks.

Models struggle to relate symptoms to diagnoses in MDDial.

MDDial will be publicly released for research use.

Abstract

Dialogue systems for Automatic Differential Diagnosis (ADD) have a wide range of real-life applications. These dialogue systems are promising for providing easy access and reducing medical costs. Building end-to-end ADD dialogue systems requires dialogue training datasets. However, to the best of our knowledge, there is no publicly available ADD dialogue dataset in English (although non-English datasets exist). Driven by this, we introduce MDDial, the first differential diagnosis dialogue dataset in English which can aid to build and evaluate end-to-end ADD dialogue systems. Additionally, earlier studies present the accuracy of diagnosis and symptoms either individually or as a combined weighted score. This method overlooks the connection between the symptoms and the diagnosis. We introduce a unified score for the ADD system that takes into account the interplay between symptoms and diagnosis. This score also indicates the system's reliability. To the end, we train two moderate-size of language models on MDDial. Our experiments suggest that while these language models can perform well on many natural language understanding tasks, including dialogue tasks in the general domain, they struggle to relate relevant symptoms and disease and thus have poor performance on MDDial. MDDial will be released publicly to aid the study of ADD dialogue research.