C3G: Learning Compact 3D Representations with 2K Gaussians

TL;DR

C3G introduces a compact, efficient 3D scene representation method using minimal Gaussians guided by self-attention, improving memory use and scene understanding in novel view synthesis.

Contribution

The paper presents a novel framework that estimates essential 3D Gaussians with learnable tokens and self-attention, reducing redundancy and enhancing multi-view feature aggregation.

Findings

Achieves high-quality scene reconstruction with fewer Gaussians.

Demonstrates superior memory efficiency over existing methods.

Effective in pose-free novel view synthesis and 3D segmentation.

Abstract

Reconstructing and understanding 3D scenes from unposed sparse views in a feed-forward manner remains as a challenging task in 3D computer vision. Recent approaches use per-pixel 3D Gaussian Splatting for reconstruction, followed by a 2D-to-3D feature lifting stage for scene understanding. However, they generate excessive redundant Gaussians, causing high memory overhead and sub-optimal multi-view feature aggregation, leading to degraded novel view synthesis and scene understanding performance. We propose C3G, a novel feed-forward framework that estimates compact 3D Gaussians only at essential spatial locations, minimizing redundancy while enabling effective feature lifting. We introduce learnable tokens that aggregate multi-view features through self-attention to guide Gaussian generation, ensuring each Gaussian integrates relevant visual features across views. We then exploit the learned attention patterns for Gaussian decoding to efficiently lift features. Extensive experiments on pose-free novel view synthesis, 3D open-vocabulary segmentation, and view-invariant feature aggregation demonstrate our approach's effectiveness. Results show that a compact yet geometrically meaningful representation is sufficient for high-quality scene reconstruction and understanding, achieving superior memory efficiency and feature fidelity compared to existing methods.