The Double Dilemma in Multi-Task Radiology Report Generation: A Gradient Dynamics Analysis and Solution

TL;DR

This paper analyzes the limitations of linear scalarization in multi-task radiology report generation using gradient dynamics, and proposes a novel optimizer, CAME-Grad, to improve clinical report quality.

Contribution

The paper introduces CAME-Grad, a conflict-averse, magnitude-enhanced optimizer that addresses the double dilemma in multi-task learning for radiology report generation.

Findings

CAME-Grad improves performance across eight RRG methods.

Achieves an average of 2.3% improvement on MIMIC-CXR.

Achieves an average of 1.9% improvement on IU X-Ray.

Abstract

While multi-task learning based automatic radiology report generation (RRG) is widely adopted to ensure clinical consistency, most focus on architectural designs yet remain limited to coarse linear scalarization strategies. These strategies cannot effectively balance the hard constraints of discriminative clinical supervision with the smoothness requirements of report generation. To address these problems, we analyze the failure mechanism of linear scalarization from the perspective of gradient dynamics, utilizing the stochastic differential equation (SDE) framework to characterize it as a "Double Dilemma" of drift term deviation and diffusion term decay. Based on this, we propose a backbone-agnostic optimizer named Conflict-Averse Magnitude-Enhanced Gradient Descent (CAME-Grad). Through conflict-averse direction rectification and magnitude-enhanced energy injection, the algorithm not only ensures geometric validity, but also avoids local optimal solutions. Then, the adaptive gradient fusion mechanism is used to establish a dynamic balance between the theoretical optimal direction and the task-specific inductive bias. Experiments show that as a universal plug-and-play optimizer, CAME-Grad brings substantial and consistent improvements across eight diverse RRG methods, elevating overall clinical efficacy performance by an average of 2.3\% on MIMIC-CXR and 1.9\% on IU X-Ray. Our code is available at https://github.com/vpsg-research/CAME-Grad.