Towards In-the-wild 3D Plane Reconstruction from a Single Image

TL;DR

ZeroPlane is a Transformer-based framework for zero-shot 3D plane detection and reconstruction from a single image, trained on a large diverse dataset, achieving superior generalization across indoor and outdoor scenes.

Contribution

Introduces ZeroPlane, a novel zero-shot 3D plane reconstruction method using a large-scale multi-domain benchmark and a disentangled, exemplar-guided learning paradigm.

Findings

Outperforms previous methods in accuracy and generalizability.

Demonstrates strong zero-shot performance on in-the-wild datasets.

Effective disentanglement of plane normal and offset improves reconstruction quality.

Abstract

3D plane reconstruction from a single image is a crucial yet challenging topic in 3D computer vision. Previous state-of-the-art (SOTA) methods have focused on training their system on a single dataset from either indoor or outdoor domain, limiting their generalizability across diverse testing data. In this work, we introduce a novel framework dubbed ZeroPlane, a Transformer-based model targeting zero-shot 3D plane detection and reconstruction from a single image, over diverse domains and environments. To enable data-driven models across multiple domains, we have curated a large-scale planar benchmark, comprising over 14 datasets and 560,000 high-resolution, dense planar annotations for diverse indoor and outdoor scenes. To address the challenge of achieving desirable planar geometry on multi-dataset training, we propose to disentangle the representation of plane normal and offset, and employ an exemplar-guided, classification-then-regression paradigm to learn plane and offset respectively. Additionally, we employ advanced backbones as image encoder, and present an effective pixel-geometry-enhanced plane embedding module to further facilitate planar reconstruction. Extensive experiments across multiple zero-shot evaluation datasets have demonstrated that our approach significantly outperforms previous methods on both reconstruction accuracy and generalizability, especially over in-the-wild data. Our code and data are available at: https://github.com/jcliu0428/ZeroPlane.