Evaluating the Pre-Consultation Ability of LLMs using Diagnostic Guidelines

TL;DR

EPAG is a new benchmark dataset and framework for assessing LLMs' pre-consultation diagnostic abilities, revealing that fine-tuned open-source models can outperform larger models in specific tasks.

Contribution

The paper introduces EPAG, a benchmark dataset and evaluation framework, and demonstrates that fine-tuned open-source models can surpass frontier LLMs in pre-consultation diagnostics.

Findings

Fine-tuned small open-source models outperform frontier LLMs in pre-consultation.

More HPI data does not necessarily improve diagnostic accuracy.

The language used in pre-consultation dialogues affects diagnostic characteristics.

Abstract

We introduce EPAG, a benchmark dataset and framework designed for Evaluating the Pre-consultation Ability of LLMs using diagnostic Guidelines. LLMs are evaluated directly through HPI-diagnostic guideline comparison and indirectly through disease diagnosis. In our experiments, we observe that small open-source models fine-tuned with a well-curated, task-specific dataset can outperform frontier LLMs in pre-consultation. Additionally, we find that increased amount of HPI (History of Present Illness) does not necessarily lead to improved diagnostic performance. Further experiments reveal that the language of pre-consultation influences the characteristics of the dialogue. By open-sourcing our dataset and evaluation pipeline on https://github.com/seemdog/EPAG, we aim to contribute to the evaluation and further development of LLM applications in real-world clinical settings.