Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imaging

TL;DR

BTB3D introduces a novel 3D tokenization method with a specialized encoder-decoder architecture, significantly improving vision-language tasks in medical imaging by maintaining anatomical detail and scalability.

Contribution

The paper presents BTB3D, a new 3D tokenization approach with a causal convolutional encoder-decoder, enabling scalable, high-resolution vision-language modeling in 3D medical imaging.

Findings

Achieves state-of-the-art report generation BLEU scores and clinical F1 improvements.

Reduces FID by 75% and halves FVD in text-to-CT synthesis.

Supports scans exceeding 300 slices without extra memory overhead.

Abstract

Recent progress in vision-language modeling for 3D medical imaging has been fueled by large-scale computed tomography (CT) corpora with paired free-text reports, stronger architectures, and powerful pretrained models. This has enabled applications such as automated report generation and text-conditioned 3D image synthesis. Yet, current approaches struggle with high-resolution, long-sequence volumes: contrastive pretraining often yields vision encoders that are misaligned with clinical language, and slice-wise tokenization blurs fine anatomy, reducing diagnostic performance on downstream tasks. We introduce BTB3D (Better Tokens for Better 3D), a causal convolutional encoder-decoder that unifies 2D and 3D training and inference while producing compact, frequency-aware volumetric tokens. A three-stage training curriculum enables (i) local reconstruction, (ii) overlapping-window tiling, and (iii) long-context decoder refinement, during which the model learns from short slice excerpts yet generalizes to scans exceeding 300 slices without additional memory overhead. BTB3D sets a new state-of-the-art on two key tasks: it improves BLEU scores and increases clinical F1 by 40% over CT2Rep, CT-CHAT, and Merlin for report generation; and it reduces FID by 75% and halves FVD compared to GenerateCT and MedSyn for text-to-CT synthesis, producing anatomically consistent 512*512*241 volumes. These results confirm that precise three-dimensional tokenization, rather than larger language backbones alone, is essential for scalable vision-language modeling in 3D medical imaging. The codebase is available at: https://github.com/ibrahimethemhamamci/BTB3D