Retrieval-Augmented Anatomical Guidance for Text-to-CT Generation

TL;DR

This paper introduces a retrieval-augmented method for text-to-CT generation that combines semantic understanding with anatomical accuracy by leveraging retrieved clinical cases as structural proxies, improving image fidelity and spatial controllability.

Contribution

It presents a novel retrieval-augmented framework that integrates anatomical guidance into text-to-CT synthesis without requiring ground-truth annotations.

Findings

Improved image fidelity and clinical consistency over text-only models.

Enables explicit spatial controllability in volumetric medical image synthesis.

Retrieval quality significantly impacts generation performance.

Abstract

Text-conditioned generative models for volumetric medical imaging provide semantic control but lack explicit anatomical guidance, often resulting in outputs that are spatially ambiguous or anatomically inconsistent. In contrast, structure-driven methods ensure strong anatomical consistency but typically assume access to ground-truth annotations, which are unavailable when the target image is to be synthesized. We propose a retrieval-augmented approach for Text-to-CT generation that integrates semantic and anatomical information under a realistic inference setting. Given a radiology report, our method retrieves a semantically related clinical case using a 3D vision-language encoder and leverages its associated anatomical annotation as a structural proxy. This proxy is injected into a text-conditioned latent diffusion model via a ControlNet branch, providing coarse anatomical guidance while maintaining semantic flexibility. Experiments on the CT-RATE dataset show that retrieval-augmented generation improves image fidelity and clinical consistency compared to text-only baselines, while additionally enabling explicit spatial controllability, a capability inherently absent in such approaches. Further analysis highlights the importance of retrieval quality, with semantically aligned proxies yielding consistent gains across all evaluation axes. This work introduces a principled and scalable mechanism to bridge semantic conditioning and anatomical plausibility in volumetric medical image synthesis. Code will be released.