CheXmix: Unified Generative Pretraining for Vision Language Models in Medical Imaging

TL;DR

CheXmix is a unified early-fusion generative model for medical imaging that improves performance on classification and report generation tasks by integrating visual and textual data effectively.

Contribution

It introduces a two-stage multimodal pretraining strategy that combines masked autoencoders with generative models, enhancing medical image understanding.

Findings

Outperforms existing models by 6.0% on AUROC at high masking ratios.

Increases image inpainting capability by 51.0%.

Outperforms CheXagent by 8.6% on AUROC and 45% on GREEN metric.

Abstract

Recent medical multimodal foundation models are built as multimodal LLMs (MLLMs) by connecting a CLIP-pretrained vision encoder to an LLM using LLaVA-style finetuning. This two-stage, decoupled approach introduces a projection layer that can distort visual features. This is especially concerning in medical imaging where subtle cues are essential for accurate diagnoses. In contrast, early-fusion generative approaches such as Chameleon eliminate the projection bottleneck by processing image and text tokens within a single unified sequence, enabling joint representation learning that leverages the inductive priors of language models. We present CheXmix, a unified early-fusion generative model trained on a large corpus of chest X-rays paired with radiology reports. We expand on Chameleon's autoregressive framework by introducing a two-stage multimodal generative pretraining strategy that combines the representational strengths of masked autoencoders with MLLMs. The resulting models are highly flexible, supporting both discriminative and generative tasks at both coarse and fine-grained scales. Our approach outperforms well-established generative models across all masking ratios by 6.0% and surpasses CheXagent by 8.6% on AUROC at high image masking ratios on the CheXpert classification task. We further inpaint images over 51.0% better than text-only generative models and outperform CheXagent by 45% on the GREEN metric for radiology report generation. These results demonstrate that CheXmix captures fine-grained information across a broad spectrum of chest X-ray tasks. Our code is at: https://github.com/StanfordMIMI/CheXmix.