Unified Medical Image Tokenizer for Autoregressive Synthesis and Understanding

TL;DR

This paper introduces MedITok, a unified medical image tokenizer trained on extensive multimodal data, enabling advanced autoregressive synthesis and understanding across diverse medical imaging modalities.

Contribution

It presents a novel two-stage training framework that leverages unpaired images and paired image-text data to build a versatile medical image tokenizer.

Findings

Achieves state-of-the-art results on over 30 benchmarks across 9 modalities.

Utilizes over 33 million images and 2 million image-text pairs for training.

Enables autoregressive modeling for diagnostic and generative medical applications.

Abstract

Autoregressive modeling has driven major advances in multimodal AI, yet its application to medical imaging remains constrained by the absence of a unified image tokenizer that simultaneously preserves fine-grained anatomical structures and rich clinical semantics across heterogeneous modalities. Existing approaches jointly optimize image reconstruction and textual semantic objectives, relying on large-scale image-caption pairs and are prone to gradient interference. This is ill-suited for the medical domain where paired data are scarce and abundant unpaired images remain unexploited. This work identifies these issues in building unified medical image tokenizers, and introduces a principled two-stage training framework using visual representation as a bridge to address them. The propose visual representation alignment stage enables the utilization of large-scale unpaired medical images to ensure reconstruction fidelity and establish foundational semantics, alleviating the interference and better preparing for the second stage where fine-grained textual semantics are injected using image-text pairs. The resulting tokenizer, MedITok, is trained on over 33 million medical images spanning 9 modalities and 2 million image-text pairs. MedITok achieves state-of-the-art performance on 30+ benchmarks spanning 9 imaging modalities and 4 task families. It further enables autoregressive modeling for diagnostic and generative applications, serving as a scalable component for future multimodal models with unified synthesis and understanding capabilities in the medical domain. Project page: https://github.com/Masaaki-75/meditok