GeoLRM: Geometry-Aware Large Reconstruction Model for High-Quality 3D Gaussian Generation

TL;DR

GeoLRM is a geometry-aware 3D reconstruction model that efficiently generates high-quality 3D assets with dense views using a novel transformer architecture and explicit geometric relationships, all within limited GPU memory.

Contribution

The paper introduces GeoLRM, a novel 3D-aware transformer that incorporates geometric relationships and deformable cross-attention for scalable, high-quality 3D Gaussian generation from limited images.

Findings

Outperforms existing models on dense view inputs

Efficiently generates 512k Gaussians with only 11 GB GPU memory

Demonstrates practical 3D generation applications

Abstract

In this work, we introduce the Geometry-Aware Large Reconstruction Model (GeoLRM), an approach which can predict high-quality assets with 512k Gaussians and 21 input images in only 11 GB GPU memory. Previous works neglect the inherent sparsity of 3D structure and do not utilize explicit geometric relationships between 3D and 2D images. This limits these methods to a low-resolution representation and makes it difficult to scale up to the dense views for better quality. GeoLRM tackles these issues by incorporating a novel 3D-aware transformer structure that directly processes 3D points and uses deformable cross-attention mechanisms to effectively integrate image features into 3D representations. We implement this solution through a two-stage pipeline: initially, a lightweight proposal network generates a sparse set of 3D anchor points from the posed image inputs; subsequently, a specialized reconstruction transformer refines the geometry and retrieves textural details. Extensive experimental results demonstrate that GeoLRM significantly outperforms existing models, especially for dense view inputs. We also demonstrate the practical applicability of our model with 3D generation tasks, showcasing its versatility and potential for broader adoption in real-world applications. The project page: https://linshan-bin.github.io/GeoLRM/.