Long-LRM: Long-sequence Large Reconstruction Model for Wide-coverage Gaussian Splats

TL;DR

Long-LRM is a fast, feed-forward 3D Gaussian reconstruction model capable of instant, high-resolution, wide-coverage scene reconstruction from multiple images, achieving comparable quality to optimization methods with significantly higher efficiency.

Contribution

It introduces a novel long-sequence transformer architecture with token merging and pruning for efficient large-scale 3D scene reconstruction from multiple images.

Findings

Achieves 800x speedup over optimization-based methods.

Handles input sizes at least 60x larger than previous feed-forward models.

Maintains comparable reconstruction quality to optimization approaches.

Abstract

We propose Long-LRM, a feed-forward 3D Gaussian reconstruction model for instant, high-resolution, 360{\deg} wide-coverage, scene-level reconstruction. Specifically, it takes in 32 input images at a resolution of 960x540 and produces the Gaussian reconstruction in just 1 second on a single A100 GPU. To handle the long sequence of 250K tokens brought by the large input size, Long-LRM features a mixture of the recent Mamba2 blocks and the classical transformer blocks, enhanced by a light-weight token merging module and Gaussian pruning steps that balance between quality and efficiency. We evaluate Long-LRM on the large-scale DL3DV benchmark and Tanks&Temples, demonstrating reconstruction quality comparable to the optimization-based methods while achieving an 800x speedup w.r.t. the optimization-based approaches and an input size at least 60x larger than the previous feed-forward approaches. We conduct extensive ablation studies on our model design choices for both rendering quality and computation efficiency. We also explore Long-LRM's compatibility with other Gaussian variants such as 2D GS, which enhances Long-LRM's ability in geometry reconstruction. Project page: https://arthurhero.github.io/projects/llrm