S-RRG-Bench: Structured Radiology Report Generation with Fine-Grained Evaluation Framework

TL;DR

This paper introduces S-RRG-Bench, a framework for generating structured radiology reports from chest X-rays, including a new dataset, a model trained on it, and a specialized evaluation metric that emphasizes clinical detail accuracy.

Contribution

The work presents a new dataset, a model for structured report generation, and a clinically meaningful evaluation metric tailored for radiology report quality assessment.

Findings

The dataset MIMIC-STRUC effectively captures clinically relevant details.

The LLM-based model generates high-quality, standardized reports.

The S-Score metric correlates strongly with human judgment of report quality.

Abstract

Radiology report generation (RRG) for diagnostic images, such as chest X-rays, plays a pivotal role in both clinical practice and AI. Traditional free-text reports suffer from redundancy and inconsistent language, complicating the extraction of critical clinical details. Structured radiology report generation (S-RRG) offers a promising solution by organizing information into standardized, concise formats. However, existing approaches often rely on classification or visual question answering (VQA) pipelines that require predefined label sets and produce only fragmented outputs. Template-based approaches, which generate reports by replacing keywords within fixed sentence patterns, further compromise expressiveness and often omit clinically important details. In this work, we present a novel approach to S-RRG that includes dataset construction, model training, and the introduction of a new evaluation framework. We first create a robust chest X-ray dataset (MIMIC-STRUC) that includes disease names, severity levels, probabilities, and anatomical locations, ensuring that the dataset is both clinically relevant and well-structured. We train an LLM-based model to generate standardized, high-quality reports. To assess the generated reports, we propose a specialized evaluation metric (S-Score) that not only measures disease prediction accuracy but also evaluates the precision of disease-specific details, thus offering a clinically meaningful metric for report quality that focuses on elements critical to clinical decision-making and demonstrates a stronger alignment with human assessments. Our approach highlights the effectiveness of structured reports and the importance of a tailored evaluation metric for S-RRG, providing a more clinically relevant measure of report quality.