MedErrBench: A Fine-Grained Multilingual Benchmark for Medical Error Detection and Correction with Clinical Expert Annotations

TL;DR

MedErrBench is a comprehensive multilingual benchmark for detecting, localizing, and correcting medical errors in clinical texts, developed with expert annotations across English, Arabic, and Chinese to improve AI healthcare safety.

Contribution

This paper introduces MedErrBench, the first multilingual clinical error detection and correction benchmark with expert annotations, covering diverse languages and error types to advance clinical NLP evaluation.

Findings

Significant performance gaps in non-English clinical texts

Language-specific models outperform general models in error tasks

Benchmark promotes development of safer, multilingual healthcare AI systems

Abstract

Inaccuracies in existing or generated clinical text may lead to serious adverse consequences, especially if it is a misdiagnosis or incorrect treatment suggestion. With Large Language Models (LLMs) increasingly being used across diverse healthcare applications, comprehensive evaluation through dedicated benchmarks is crucial. However, such datasets remain scarce, especially across diverse languages and contexts. In this paper, we introduce MedErrBench, the first multilingual benchmark for error detection, localization, and correction, developed under the guidance of experienced clinicians. Based on an expanded taxonomy of ten common error types, MedErrBench covers English, Arabic and Chinese, with natural clinical cases annotated and reviewed by domain experts. We assessed the performance of a range of general-purpose, language-specific, and medical-domain language models across all three tasks. Our results reveal notable performance gaps, particularly in non-English settings, highlighting the need for clinically grounded, language-aware systems. By making MedErrBench and our evaluation protocols publicly-available, we aim to advance multilingual clinical NLP to promote safer and more equitable AI-based healthcare globally. The dataset is available in the supplementary material. An anonymized version of the dataset is available at: https://github.com/congboma/MedErrBench.