Chorus: Multi-Teacher Pretraining for Holistic 3D Gaussian Scene Encoding

TL;DR

Chorus introduces a multi-teacher pretraining framework for 3D Gaussian scene encoding, leveraging diverse signals from foundation models to improve various 3D understanding tasks with fewer scenes.

Contribution

It presents a novel multi-teacher pretraining approach for 3D Gaussian scene encoders, enabling effective transfer and outperforms point-cloud baselines with significantly less data.

Findings

Chorus achieves strong transfer performance on multiple 3D tasks.

Pretraining with Chorus outperforms point-cloud baselines using 39.9 times fewer scenes.

The render-and-distill method facilitates effective out-of-domain finetuning.

Abstract

While 3DGS has emerged as a high-fidelity scene representation, encoding rich, general-purpose features directly from its primitives remains under-explored. We address this gap by introducing Chorus, a multi-teacher pretraining framework that learns a holistic feed-forward 3D Gaussian Splatting (3DGS) scene encoder by distilling complementary signals from 2D foundation models. Chorus employs a shared 3D encoder and teacher-specific projectors to learn from language-aligned, generalist, and object-aware teachers, encouraging a shared embedding space that captures signals from high-level semantics to fine-grained structure. We evaluate Chorus on a wide range of tasks: open-vocabulary semantic and instance segmentation, linear and decoder probing, data-efficient supervision, as well as LLM-based Q&A. Besides 3DGS, we also test Chorus on several benchmarks that only support point clouds by pretraining a variant using only Gaussian centers, colors, and estimated normals. Surprisingly, this encoder shows strong transfer and outperforms the point-cloud baseline while using 39.9 times fewer training scenes. Finally, we propose a render-and-distill adaptation that facilitates out-of-domain finetuning.