LLM-Mini-CEX: Automatic Evaluation of Large Language Model for Diagnostic Conversation

TL;DR

This paper introduces LLM-Mini-CEX, an automatic, comprehensive evaluation framework for medical diagnostic LLMs, utilizing a patient simulator and ChatGPT to automate assessment and address labor-intensive evaluation processes.

Contribution

It proposes a novel, unified evaluation criterion for medical LLMs and automates the evaluation process using ChatGPT and a patient simulator, reducing manual effort.

Findings

LLM-specific Mini-CEX effectively evaluates diagnostic capabilities.

ChatGPT can replace manual evaluation for dialogue quality.

Automated evaluation is reproducible and efficient.

Abstract

There is an increasing interest in developing LLMs for medical diagnosis to improve diagnosis efficiency. Despite their alluring technological potential, there is no unified and comprehensive evaluation criterion, leading to the inability to evaluate the quality and potential risks of medical LLMs, further hindering the application of LLMs in medical treatment scenarios. Besides, current evaluations heavily rely on labor-intensive interactions with LLMs to obtain diagnostic dialogues and human evaluation on the quality of diagnosis dialogue. To tackle the lack of unified and comprehensive evaluation criterion, we first initially establish an evaluation criterion, termed LLM-specific Mini-CEX to assess the diagnostic capabilities of LLMs effectively, based on original Mini-CEX. To address the labor-intensive interaction problem, we develop a patient simulator to engage in automatic conversations with LLMs, and utilize ChatGPT for evaluating diagnosis dialogues automatically. Experimental results show that the LLM-specific Mini-CEX is adequate and necessary to evaluate medical diagnosis dialogue. Besides, ChatGPT can replace manual evaluation on the metrics of humanistic qualities and provides reproducible and automated comparisons between different LLMs.