MedErr-CT: A Visual Question Answering Benchmark for Identifying and Correcting Errors in CT Reports

TL;DR

MedErr-CT is a new benchmark designed to evaluate medical multimodal large language models' ability to identify and correct errors in CT reports, addressing a gap in clinical relevance and expert-level understanding.

Contribution

It introduces a comprehensive VQA benchmark with diverse error categories and task levels for assessing and improving medical MLLMs' diagnostic accuracy.

Findings

Significant variation in model performance across error types

Benchmark reveals strengths and weaknesses of current medical MLLMs

Provides a foundation for developing more reliable clinical AI tools

Abstract

Computed Tomography (CT) plays a crucial role in clinical diagnosis, but the growing demand for CT examinations has raised concerns about diagnostic errors. While Multimodal Large Language Models (MLLMs) demonstrate promising comprehension of medical knowledge, their tendency to produce inaccurate information highlights the need for rigorous validation. However, existing medical visual question answering (VQA) benchmarks primarily focus on simple visual recognition tasks, lacking clinical relevance and failing to assess expert-level knowledge. We introduce MedErr-CT, a novel benchmark for evaluating medical MLLMs' ability to identify and correct errors in CT reports through a VQA framework. The benchmark includes six error categories - four vision-centric errors (Omission, Insertion, Direction, Size) and two lexical error types (Unit, Typo) - and is organized into three task levels: classification, detection, and correction. Using this benchmark, we quantitatively assess the performance of state-of-the-art 3D medical MLLMs, revealing substantial variation in their capabilities across different error types. Our benchmark contributes to the development of more reliable and clinically applicable MLLMs, ultimately helping reduce diagnostic errors and improve accuracy in clinical practice. The code and datasets are available at https://github.com/babbu3682/MedErr-CT.