Local Patches Meet Global Context: Scalable 3D Diffusion Priors for Computed Tomography Reconstruction

TL;DR

This paper introduces a scalable 3D diffusion prior model that leverages local patches and global context to improve high-resolution 3D CT reconstruction, overcoming computational challenges of traditional methods.

Contribution

The study proposes a novel 3D patch-based diffusion model that learns from limited data, enabling efficient high-resolution 3D image generation and inverse problem solving.

Findings

Outperforms state-of-the-art methods in accuracy and efficiency.

Achieves high-resolution 3D reconstruction of 512×512×256 in about 20 minutes.

Effectively leverages local and global information for 3D image quality.

Abstract

Diffusion models learn strong image priors that can be leveraged to solve inverse problems like medical image reconstruction. However, for real-world applications such as 3D Computed Tomography (CT) imaging, directly training diffusion models on 3D data presents significant challenges due to the high computational demands of extensive GPU resources and large-scale datasets. Existing works mostly reuse 2D diffusion priors to address 3D inverse problems, but fail to fully realize and leverage the generative capacity of diffusion models for high-dimensional data. In this study, we propose a novel 3D patch-based diffusion model that can learn a fully 3D diffusion prior from limited data, enabling scalable generation of high-resolution 3D images. Our core idea is to learn the prior of 3D patches to achieve scalable efficiency, while coupling local and global information to guarantee high-quality 3D image generation, by modeling the joint distribution of position-aware 3D local patches and downsampled 3D volume as global context. Our approach not only enables high-quality 3D generation, but also offers an unprecedentedly efficient and accurate solution to high-resolution 3D inverse problems. Experiments on 3D CT reconstruction across multiple datasets show that our method outperforms state-of-the-art methods in both performance and efficiency, notably achieving high-resolution 3D reconstruction of $512 \times 512 \times 256$ ($\sim$20 mins).