4-LEGS: 4D Language Embedded Gaussian Splatting

TL;DR

This paper introduces a 4D neural representation method that combines language and spatio-temporal modeling using Gaussian Splatting, enabling interactive text-based localization of events in 3D videos.

Contribution

It presents a novel approach to connect language with dynamic 3D scene representations by lifting features into 4D space using Gaussian Splatting.

Findings

Enables spatiotemporal localization of events from text prompts

Demonstrates effectiveness on 3D video datasets of people and animals

Provides an interactive interface for scene understanding

Abstract

The emergence of neural representations has revolutionized our means for digitally viewing a wide range of 3D scenes, enabling the synthesis of photorealistic images rendered from novel views. Recently, several techniques have been proposed for connecting these low-level representations with the high-level semantics understanding embodied within the scene. These methods elevate the rich semantic understanding from 2D imagery to 3D representations, distilling high-dimensional spatial features onto 3D space. In our work, we are interested in connecting language with a dynamic modeling of the world. We show how to lift spatio-temporal features to a 4D representation based on 3D Gaussian Splatting. This enables an interactive interface where the user can spatiotemporally localize events in the video from text prompts. We demonstrate our system on public 3D video datasets of people and animals performing various actions.