Deep Diffusion Image Prior for Efficient OOD Adaptation in 3D Inverse Problems

TL;DR

This paper introduces DDIP and D3IP, efficient methods for adapting diffusion priors to 3D inverse problems, enabling high-quality reconstructions even with limited or mismatched training data.

Contribution

It presents a formal connection between diffusion priors and deep image priors, along with a fast adaptation method for 3D inverse problems and demonstrates meta-learning enhancements.

Findings

D3IP accelerates DDIP by orders of magnitude.

The method achieves superior 3D reconstructions from limited training data.

Meta-learning further improves performance.

Abstract

Recent inverse problem solvers that leverage generative diffusion priors have garnered significant attention due to their exceptional quality. However, adaptation of the prior is necessary when there exists a discrepancy between the training and testing distributions. In this work, we propose deep diffusion image prior (DDIP), which generalizes the recent adaptation method of SCD by introducing a formal connection to the deep image prior. Under this framework, we propose an efficient adaptation method dubbed D3IP, specified for 3D measurements, which accelerates DDIP by orders of magnitude while achieving superior performance. D3IP enables seamless integration of 3D inverse solvers and thus leads to coherent 3D reconstruction. Moreover, we show that meta-learning techniques can also be applied to yield even better performance. We show that our method is capable of solving diverse 3D reconstructive tasks from the generative prior trained only with phantom images that are vastly different from the training set, opening up new opportunities of applying diffusion inverse solvers even when training with gold standard data is impossible. Code: https://github.com/HJ-harry/DDIP3D