Exploring the Inquiry-Diagnosis Relationship with Advanced Patient Simulators

TL;DR

This paper introduces a novel patient simulator trained on real doctor-patient dialogues, enabling more accurate evaluation of diagnostic models and revealing the critical impact of inquiry quality on diagnosis effectiveness.

Contribution

It presents a new dialogue strategy-based patient simulator that improves realism and explores the relationship between inquiry and diagnosis in medical AI models.

Findings

Inquiry quality significantly affects diagnostic accuracy.

Models show varying inquiry performance, impacting overall diagnosis.

Inquiry and diagnosis adhere to Liebig's law, limiting each other's effectiveness.

Abstract

Recently, large language models have shown great potential to transform online medical consultation. Despite this, most research targets improving diagnostic accuracy with ample information, often overlooking the inquiry phase. Some studies try to evaluate or refine doctor models by using prompt-engineered patient agents. However, prompt engineering alone falls short in accurately simulating real patients. We need to explore new paradigms for patient simulation. Furthermore, the relationship between inquiry and diagnosis remains unexplored. This paper extracts dialogue strategies from real doctor-patient conversations to guide the training of a patient simulator. Our simulator shows higher anthropomorphism and lower hallucination rates, using dynamic dialogue strategies. This innovation offers a more accurate evaluation of diagnostic models and generates realistic synthetic data. We conduct extensive experiments on the relationship between inquiry and diagnosis, showing they adhere to Liebig's law: poor inquiry limits diagnosis effectiveness, regardless of diagnostic skill, and vice versa. The experiments also reveal substantial differences in inquiry performance among models. To delve into this phenomenon, the inquiry process is categorized into four distinct types. Analyzing the distribution of inquiries across these types helps explain the performance differences. The weights of our patient simulator are available https://github.com/PatientSimulator/PatientSimulator.