An X-Ray Is Worth 15 Features: Sparse Autoencoders for Interpretable Radiology Report Generation

TL;DR

SAE-Rad introduces sparse autoencoders to decompose vision transformer features into interpretable radiology report components, achieving competitive results with less computational cost and improved interpretability.

Contribution

This work is the first to apply mechanistic interpretability techniques to a multi-modal reasoning task in radiology report generation.

Findings

Achieves competitive radiology metrics on MIMIC-CXR

Uses significantly less training resources

Learns meaningful visual concepts aligned with expert interpretations

Abstract

Radiological services are experiencing unprecedented demand, leading to increased interest in automating radiology report generation. Existing Vision-Language Models (VLMs) suffer from hallucinations, lack interpretability, and require expensive fine-tuning. We introduce SAE-Rad, which uses sparse autoencoders (SAEs) to decompose latent representations from a pre-trained vision transformer into human-interpretable features. Our hybrid architecture combines state-of-the-art SAE advancements, achieving accurate latent reconstructions while maintaining sparsity. Using an off-the-shelf language model, we distil ground-truth reports into radiological descriptions for each SAE feature, which we then compile into a full report for each image, eliminating the need for fine-tuning large models for this task. To the best of our knowledge, SAE-Rad represents the first instance of using mechanistic interpretability techniques explicitly for a downstream multi-modal reasoning task. On the MIMIC-CXR dataset, SAE-Rad achieves competitive radiology-specific metrics compared to state-of-the-art models while using significantly fewer computational resources for training. Qualitative analysis reveals that SAE-Rad learns meaningful visual concepts and generates reports aligning closely with expert interpretations. Our results suggest that SAEs can enhance multimodal reasoning in healthcare, providing a more interpretable alternative to existing VLMs.