BuildAnyPoint: 3D Building Structured Abstraction from Diverse Point Clouds

TL;DR

BuildAnyPoint is a new generative framework that reconstructs structured 3D building models from diverse point clouds using a diffusion transformer and autoregressive mesh generation, improving accuracy and distribution quality.

Contribution

It introduces a novel combination of diffusion models and autoregressive transformers for 3D building reconstruction from varied point clouds, a new approach in this domain.

Findings

Outperforms prior building abstraction methods qualitatively and quantitatively.

Achieves improved surface accuracy and distribution uniformity in point cloud completion.

Demonstrates strong performance on building point cloud benchmarks.

Abstract

We introduce BuildAnyPoint, a novel generative framework for structured 3D building reconstruction from point clouds with diverse distributions, such as those captured by airborne LiDAR and Structure-from-Motion. To recover artist-created building abstraction in this highly underconstrained setting, we capitalize on the role of explicit 3D generative priors in autoregressive mesh generation. Specifically, we design a Loosely Cascaded Diffusion Transformer (Loca-DiT) that initially recovers the underlying distribution from noisy or sparse points, followed by autoregressively encapsulating them into compact meshes. We first formulate distribution recovery as a conditional generation task by training latent diffusion models conditioned on input point clouds, and then tailor a decoder-only transformer for conditional autoregressive mesh generation based on the recovered point clouds. Our method delivers substantial qualitative and quantitative improvements over prior building abstraction methods. Furthermore, the effectiveness of our approach is evidenced by the strong performance of its recovered point clouds on building point cloud completion benchmarks, which exhibit improved surface accuracy and distribution uniformity.