MetaVoxel: Joint Diffusion Modeling of Imaging and Clinical Metadata

TL;DR

MetaVoxel introduces a unified diffusion model that jointly learns imaging and clinical data distributions, enabling versatile tasks like image synthesis and demographic prediction without task-specific retraining.

Contribution

It presents the first joint diffusion framework for medical imaging and metadata, supporting flexible zero-shot inference across multiple tasks.

Findings

Achieves comparable performance to task-specific models in image generation, age, and sex prediction.

Supports flexible inference with arbitrary input subsets.

Demonstrates potential for unified medical AI modeling.

Abstract

Modern deep learning methods have achieved impressive results across tasks from disease classification, estimating continuous biomarkers, to generating realistic medical images. Most of these approaches are trained to model conditional distributions defined by a specific predictive direction with a specific set of input variables. We introduce MetaVoxel, a generative joint diffusion modeling framework that models the joint distribution over imaging data and clinical metadata by learning a single diffusion process spanning all variables. By capturing the joint distribution, MetaVoxel unifies tasks that traditionally require separate conditional models and supports flexible zero-shot inference using arbitrary subsets of inputs without task-specific retraining. Using more than 10,000 T1-weighted MRI scans paired with clinical metadata from nine datasets, we show that a single MetaVoxel model can perform image generation, age estimation, and sex prediction, achieving performance comparable to established task-specific baselines. Additional experiments highlight its capabilities for flexible inference. Together, these findings demonstrate that joint multimodal diffusion offers a promising direction for unifying medical AI models and enabling broader clinical applicability.