Feature Splatting: Language-Driven Physics-Based Scene Synthesis and Editing

TL;DR

Feature Splatting introduces a unified method for scene synthesis and editing that combines physics-based dynamics with vision-language grounded semantics using 3D Gaussian primitives.

Contribution

It presents a novel approach to incorporate natural language grounded semantics and physics-based properties into 3D scene representations using feature-splatting.

Findings

Enables semi-automatic scene decomposition with text queries.

Synthesizes physics-based object dynamics automatically from static scenes.

Demonstrates the versatility of 3D Gaussians for appearance, geometry, and semantics.

Abstract

Scene representations using 3D Gaussian primitives have produced excellent results in modeling the appearance of static and dynamic 3D scenes. Many graphics applications, however, demand the ability to manipulate both the appearance and the physical properties of objects. We introduce Feature Splatting, an approach that unifies physics-based dynamic scene synthesis with rich semantics from vision language foundation models that are grounded by natural language. Our first contribution is a way to distill high-quality, object-centric vision-language features into 3D Gaussians, that enables semi-automatic scene decomposition using text queries. Our second contribution is a way to synthesize physics-based dynamics from an otherwise static scene using a particle-based simulator, in which material properties are assigned automatically via text queries. We ablate key techniques used in this pipeline, to illustrate the challenge and opportunities in using feature-carrying 3D Gaussians as a unified format for appearance, geometry, material properties and semantics grounded on natural language. Project website: https://feature-splatting.github.io/