ArtiFixer: Enhancing and Extending 3D Reconstruction with Auto-Regressive Diffusion Models

TL;DR

ArtiFixer introduces a two-stage auto-regressive diffusion-based approach that significantly improves 3D scene reconstruction quality and scalability, enabling high-fidelity novel view synthesis even in unobserved regions.

Contribution

The paper presents a novel bidirectional generative model with an opacity mixing strategy, distilled into a causal auto-regressive model for efficient, high-quality 3D reconstruction.

Findings

Outperforms existing methods by 1-3 dB PSNR on benchmark datasets.

Generates hundreds of views in a single pass with high consistency.

Successfully reconstructs scenes where prior approaches fail.

Abstract

Per-scene optimization methods such as 3D Gaussian Splatting provide state-of-the-art novel view synthesis quality but extrapolate poorly to under-observed areas. Methods that leverage generative priors to correct artifacts in these areas hold promise but currently suffer from two shortcomings. The first is scalability, as existing methods use image diffusion models or bidirectional video models that are limited in the number of views they can generate in a single pass (and thus require a costly iterative distillation process for consistency). The second is quality itself, as generators used in prior work tend to produce outputs that are inconsistent with existing scene content and fail entirely in completely unobserved regions. To solve these, we propose a two-stage pipeline that leverages two key insights. First, we train a powerful bidirectional generative model with a novel opacity mixing strategy that encourages consistency with existing observations while retaining the model's ability to extrapolate novel content in unseen areas. Second, we distill it into a causal auto-regressive model that generates hundreds of frames in a single pass. This model can directly produce novel views or serve as pseudo-supervision to improve the underlying 3D representation in a simple and highly efficient manner. We evaluate our method extensively and demonstrate that it can generate plausible reconstructions in scenarios where existing approaches fail completely. When measured on commonly benchmarked datasets, we outperform all existing baselines by a wide margin, exceeding prior state-of-the-art methods by 1-3 dB PSNR.