Importance of Prompt Optimisation for Error Detection in Medical Notes Using Language Models

TL;DR

This paper demonstrates that prompt optimisation significantly enhances the accuracy of language models in detecting errors in medical notes, approaching expert-level performance and outperforming baseline models.

Contribution

It introduces a genetic-pareto based prompt optimisation method that improves error detection accuracy in medical text using various language models.

Findings

Prompt optimisation boosts GPT-5 error detection accuracy from 0.669 to 0.785.

Qwen3-32B accuracy improves from 0.578 to 0.690 with prompt optimisation.

Achieves state-of-the-art results on the MEDEC benchmark dataset.

Abstract

Errors in medical text can cause delays or even result in incorrect treatment for patients. Recently, language models have shown promise in their ability to automatically detect errors in medical text, an ability that has the opportunity to significantly benefit healthcare systems. In this paper, we explore the importance of prompt optimisation for small and large language models when applied to the task of error detection. We perform rigorous experiments and analysis across frontier language models and open-source language models. We show that automatic prompt optimisation with Genetic-Pareto (GEPA) improves error detection over the baseline accuracy performance from 0.669 to 0.785 with GPT-5 and 0.578 to 0.690 with Qwen3-32B, approaching the performance of medical doctors and achieving state-of-the-art performance on the MEDEC benchmark dataset. Code available on GitHub: https://github.com/CraigMyles/clinical-note-error-detection