LUDVIG: Learning-Free Uplifting of 2D Visual Features to Gaussian Splatting Scenes

TL;DR

This paper introduces LUDVIG, a learning-free, efficient method to uplift 2D vision features into 3D Gaussian Splatting representations, enabling fast and effective 3D scene understanding without extensive training.

Contribution

LUDVIG presents a novel, training-free approach using feature aggregation and graph diffusion to convert 2D features into 3D scenes, outperforming traditional reconstruction-based methods in speed and comparable accuracy.

Findings

Achieves competitive segmentation with DINOv2 features without training on segmentation masks.

Demonstrates strong open-vocabulary object segmentation with CLIP features.

Provides significant speed-ups over traditional 3D reconstruction methods.

Abstract

We address the problem of extending the capabilities of vision foundation models such as DINO, SAM, and CLIP, to 3D tasks. Specifically, we introduce a novel method to uplift 2D image features into Gaussian Splatting representations of 3D scenes. Unlike traditional approaches that rely on minimizing a reconstruction loss, our method employs a simpler and more efficient feature aggregation technique, augmented by a graph diffusion mechanism. Graph diffusion refines 3D features, such as coarse segmentation masks, by leveraging 3D geometry and pairwise similarities induced by DINOv2. Our approach achieves performance comparable to the state of the art on multiple downstream tasks while delivering significant speed-ups. Notably, we obtain competitive segmentation results using only generic DINOv2 features, despite DINOv2 not being trained on millions of annotated segmentation masks like SAM. When applied to CLIP features, our method demonstrates strong performance in open-vocabulary object segmentation tasks, highlighting the versatility of our approach.