A Unified Framework for Multimodal Image Reconstruction and Synthesis using Denoising Diffusion Models

TL;DR

This paper presents Any2all, a unified diffusion-based framework that simplifies multimodal image reconstruction and synthesis by treating all tasks as inpainting, enabling flexible, high-quality results across various imaging modalities.

Contribution

The paper introduces a single diffusion model that unifies multiple multimodal imaging tasks, reducing complexity and improving performance over specialized methods.

Findings

Achieves high-quality multimodal reconstruction and synthesis.

Outperforms specialized methods in perceptual quality.

Demonstrates versatility across PET/MR/CT datasets.

Abstract

Image reconstruction and image synthesis are important for handling incomplete multimodal imaging data, but existing methods require various task-specific models, complicating training and deployment workflows. We introduce Any2all, a unified framework that addresses this limitation by formulating these disparate tasks as a single virtual inpainting problem. We train a single, unconditional diffusion model on the complete multimodal data stack. This model is then adapted at inference time to ``inpaint'' all target modalities from any combination of inputs of available clean images or noisy measurements. We validated Any2all on a PET/MR/CT brain dataset. Our results show that Any2all can achieve excellent performance on both multimodal reconstruction and synthesis tasks, consistently yielding images with competitive distortion-based performance and superior perceptual quality over specialized methods.