Leveraging Multimodal Diffusion Models to Accelerate Imaging with Side Information

TL;DR

This paper introduces a multimodal diffusion model framework that leverages cheaper auxiliary data to reduce the number of measurements needed for high-quality imaging in scientific applications, especially materials science.

Contribution

The authors develop a novel training framework for multimodal diffusion models that transforms complex inverse problems with black-box models into simple inpainting tasks, enabling efficient imaging with less data.

Findings

Achieves superior image reconstruction using side information.

Requires significantly less data from expensive microscopy.

Demonstrates feasibility on materials imagery data.

Abstract

Diffusion models have found phenomenal success as expressive priors for solving inverse problems, but their extension beyond natural images to more structured scientific domains remains limited. Motivated by applications in materials science, we aim to reduce the number of measurements required from an expensive imaging modality of interest, by leveraging side information from an auxiliary modality that is much cheaper to obtain. To deal with the non-differentiable and black-box nature of the forward model, we propose a framework to train a multimodal diffusion model over the joint modalities, turning inverse problems with black-box forward models into simple linear inpainting problems. Numerically, we demonstrate the feasibility of training diffusion models over materials imagery data, and show that our approach achieves superior image reconstruction by leveraging the available side information, requiring significantly less amount of data from the expensive microscopy modality.