3D-Fixer: Coarse-to-Fine In-place Completion for 3D Scenes from a Single Image

TL;DR

3D-Fixer introduces a coarse-to-fine in-place completion method for 3D scene generation from a single image, improving accuracy and efficiency by leveraging a novel dataset and a dual-branch network.

Contribution

It proposes a new in-place completion paradigm with a coarse-to-fine scheme and a dual-branch network, addressing boundary ambiguity and data scarcity in 3D scene reconstruction.

Findings

Achieves state-of-the-art geometric accuracy in 3D scene completion.

Outperforms baselines like MIDI and Gen3DSR in experiments.

Maintains efficiency comparable to diffusion-based methods.

Abstract

Compositional 3D scene generation from a single view requires the simultaneous recovery of scene layout and 3D assets. Existing approaches mainly fall into two categories: feed-forward generation methods and per-instance generation methods. The former directly predict 3D assets with explicit 6DoF poses through efficient network inference, but they generalize poorly to complex scenes. The latter improve generalization through a divide-and-conquer strategy, but suffer from time-consuming pose optimization. To bridge this gap, we introduce 3D-Fixer, a novel in-place completion paradigm. Specifically, 3D-Fixer extends 3D object generative priors to generate complete 3D assets conditioned on the partially visible point cloud at the original locations, which are cropped from the fragmented geometry obtained from the geometry estimation methods. Unlike prior works that require explicit pose alignment, 3D-Fixer uses fragmented geometry as a spatial anchor to preserve layout fidelity. At its core, we propose a coarse-to-fine generation scheme to resolve boundary ambiguity under occlusion, supported by a dual-branch conditioning network and an Occlusion-Robust Feature Alignment (ORFA) strategy for stable training. Furthermore, to address the data scarcity bottleneck, we present ARSG-110K, the largest scene-level dataset to date, comprising over 110K diverse scenes and 3M annotated images with high-fidelity 3D ground truth. Extensive experiments show that 3D-Fixer achieves state-of-the-art geometric accuracy, which significantly outperforms baselines such as MIDI and Gen3DSR, while maintaining the efficiency of the diffusion process. Code and data will be publicly available at https://zx-yin.github.io/3dfixer.