Feat2GS: Probing Visual Foundation Models with Gaussian Splatting

TL;DR

This paper introduces Feat2GS, a novel framework that probes the 3D understanding of visual foundation models using Gaussian splatting, enabling texture and geometry analysis without 3D data, and advancing view synthesis capabilities.

Contribution

We propose Feat2GS, a new method for assessing 3D awareness in VFMs through Gaussian splatting, which does not require 3D ground-truth data and allows separate analysis of geometry and texture.

Findings

Feats2GS effectively probes 3D awareness in VFMs.

Our variants achieve state-of-the-art results on diverse datasets.

The method enables novel view synthesis without 3D supervision.

Abstract

Given that visual foundation models (VFMs) are trained on extensive datasets but often limited to 2D images, a natural question arises: how well do they understand the 3D world? With the differences in architecture and training protocols (i.e., objectives, proxy tasks), a unified framework to fairly and comprehensively probe their 3D awareness is urgently needed. Existing works on 3D probing suggest single-view 2.5D estimation (e.g., depth and normal) or two-view sparse 2D correspondence (e.g., matching and tracking). Unfortunately, these tasks ignore texture awareness, and require 3D data as ground-truth, which limits the scale and diversity of their evaluation set. To address these issues, we introduce Feat2GS, which readout 3D Gaussians attributes from VFM features extracted from unposed images. This allows us to probe 3D awareness for geometry and texture via novel view synthesis, without requiring 3D data. Additionally, the disentanglement of 3DGS parameters - geometry ($\boldsymbol{x}$, $\alpha$, $\Sigma$) and texture ($\boldsymbol{c}$) - enables separate analysis of texture and geometry awareness. Under Feat2GS, we conduct extensive experiments to probe the 3D awareness of several VFMs, and investigate the ingredients that lead to a 3D aware VFM. Building on these findings, we develop several variants that achieve state-of-the-art across diverse datasets. This makes Feat2GS useful for probing VFMs, and as a simple-yet-effective baseline for novel-view synthesis. Code and data are available at https://fanegg.github.io/Feat2GS/.