Enhancing Reinforcement Learning for Radiology Report Generation with Evidence-aware Rewards and Self-correcting Preference Learning

TL;DR

This paper introduces ESC-RL, a reinforcement learning framework for radiology report generation that uses evidence-aware rewards and self-correcting preference learning to improve clinical faithfulness and alignment.

Contribution

It proposes a novel evidence-aware reward mechanism and a self-correcting preference learning strategy to enhance clinical accuracy and continual self-improvement in report generation.

Findings

Achieves state-of-the-art results on two chest X-ray datasets.

Demonstrates improved clinical faithfulness and disease alignment.

Supports continual self-improvement during training.

Abstract

Recent reinforcement learning (RL) approaches have advanced radiology report generation (RRG), yet two core limitations persist: (1) report-level rewards offer limited evidence-grounded guidance for clinical faithfulness; and (2) current methods lack an explicit self-improving mechanism to align with clinical preference. We introduce clinically aligned Evidence-aware Self-Correcting Reinforcement Learning (ESC-RL), comprising two key components. First, a Group-wise Evidence-aware Alignment Reward (GEAR) delivers group-wise, evidence-aware feedback. GEAR reinforces consistent grounding for true positives, recovers missed findings for false negatives, and suppresses unsupported content for false positives. Second, a Self-correcting Preference Learning (SPL) strategy automatically constructs a reliable, disease-aware preference dataset from multiple noisy observations and leverages an LLM to synthesize refined reports without human supervision. ESC-RL promotes clinically faithful, disease-aligned reward and supports continual self-improvement during training. Extensive experiments on two public chest X-ray datasets demonstrate consistent gains and state-of-the-art performance.