Unaligned 2D to 3D Translation with Conditional Vector-Quantized Code Diffusion using Transformers

TL;DR

This paper introduces a novel 2D to 3D synthesis method using conditional diffusion with vector-quantized codes, enabling high-resolution 3D image generation from limited 2D views, outperforming existing approaches.

Contribution

It presents a new diffusion-based framework operating in a code space for effective 2D to 3D translation, addressing domain gap and geometric misalignment issues.

Findings

Achieves state-of-the-art results on complex volumetric datasets.

Demonstrates high fidelity and coverage in 3D reconstructions.

Outperforms specialized methods across multiple evaluation metrics.

Abstract

Generating 3D images of complex objects conditionally from a few 2D views is a difficult synthesis problem, compounded by issues such as domain gap and geometric misalignment. For instance, a unified framework such as Generative Adversarial Networks cannot achieve this unless they explicitly define both a domain-invariant and geometric-invariant joint latent distribution, whereas Neural Radiance Fields are generally unable to handle both issues as they optimize at the pixel level. By contrast, we propose a simple and novel 2D to 3D synthesis approach based on conditional diffusion with vector-quantized codes. Operating in an information-rich code space enables high-resolution 3D synthesis via full-coverage attention across the views. Specifically, we generate the 3D codes (e.g. for CT images) conditional on previously generated 3D codes and the entire codebook of two 2D views (e.g. 2D X-rays). Qualitative and quantitative results demonstrate state-of-the-art performance over specialized methods across varied evaluation criteria, including fidelity metrics such as density, coverage, and distortion metrics for two complex volumetric imagery datasets from in real-world scenarios.