CT-Bench: A Benchmark for Multimodal Lesion Understanding in Computed Tomography

TL;DR

CT-Bench introduces a comprehensive multimodal benchmark dataset for lesion understanding in CT scans, enabling evaluation and improvement of AI models in lesion detection, description, and diagnosis tasks.

Contribution

This work provides the first large-scale, annotated CT lesion dataset and a multitask benchmark for evaluating multimodal AI models in medical imaging.

Findings

State-of-the-art models outperform radiologists in some tasks.

Fine-tuning improves model performance significantly.

CT-Bench facilitates comprehensive lesion analysis evaluation.

Abstract

Artificial intelligence (AI) can automatically delineate lesions on computed tomography (CT) and generate radiology report content, yet progress is limited by the scarcity of publicly available CT datasets with lesion-level annotations. To bridge this gap, we introduce CT-Bench, a first-of-its-kind benchmark dataset comprising two components: a Lesion Image and Metadata Set containing 20,335 lesions from 7,795 CT studies with bounding boxes, descriptions, and size information, and a multitask visual question answering benchmark with 2,850 QA pairs covering lesion localization, description, size estimation, and attribute categorization. Hard negative examples are included to reflect real-world diagnostic challenges. We evaluate multiple state-of-the-art multimodal models, including vision-language and medical CLIP variants, by comparing their performance to radiologist assessments, demonstrating the value of CT-Bench as a comprehensive benchmark for lesion analysis. Moreover, fine-tuning models on the Lesion Image and Metadata Set yields significant performance gains across both components, underscoring the clinical utility of CT-Bench.