Off The Grid: Detection of Primitives for Feed-Forward 3D Gaussian Splatting

TL;DR

This paper introduces an adaptive, off-the-grid primitive detection method for 3D Gaussian Splatting that improves scene quality and efficiency in real-time view synthesis.

Contribution

It proposes a novel decoder architecture that detects primitives at a sub-pixel level and an adaptive density mechanism, enhancing 3D scene generation without 3D annotations.

Findings

Achieves state-of-the-art novel view synthesis with fewer primitives.

Generates photorealistic 3D scenes in seconds.

Outperforms existing methods in quality and efficiency.

Abstract

Feed-forward 3D Gaussian Splatting (3DGS) models enable real-time scene generation but are hindered by suboptimal pixel-aligned primitive placement, which relies on a dense, rigid grid that limits both quality and efficiency. We introduce a new feed-forward architecture that detects 3D Gaussian primitives at a sub-pixel level, replacing the pixel grid with an adaptive, ``Off-The-Grid" distribution. Inspired by keypoint detection, our decoder learns to locally distribute primitives across image patches. We also provide an Adaptive Density mechanism by assigning varying number of primitives per patch based on Shannon entropy. We combine the proposed decoder with a pre-trained 3D reconstruction backbone and train them end-to-end using photometric supervision without any 3D annotation. The resulting pose-free model generates photorealistic 3DGS scenes in seconds, achieving state-of-the-art novel view synthesis for feed-forward models. It outperforms competitors while using far fewer primitives, demonstrating a more accurate and efficient allocation that captures fine details and reduces artifacts. Project page: https://arthurmoreau.github.io/OffTheGrid/.