TRASE: Tracking-free 4D Segmentation and Editing

TL;DR

TRASE is a novel tracking-free 4D segmentation method that enables semantic understanding and editing of dynamic 3D scenes using weakly-supervised learning and contrastive features, advancing scene reconstruction and interaction.

Contribution

It introduces a weakly-supervised 4D segmentation approach that produces semantically coherent features for dynamic scenes, facilitating fast editing without tracking.

Findings

Achieves state-of-the-art segmentation performance on five benchmarks.

Enables effective scene editing tasks like object removal and style transfer.

Demonstrates robustness across unseen viewpoints.

Abstract

Understanding dynamic 3D scenes is crucial for extended reality (XR) and autonomous driving. Incorporating semantic information into 3D reconstruction enables holistic scene representations, unlocking immersive and interactive applications. To this end, we introduce TRASE, a novel tracking-free 4D segmentation method for dynamic scene understanding. TRASE learns a 4D segmentation feature field in a weakly-supervised manner, leveraging a soft-mined contrastive learning objective guided by SAM masks. The resulting feature space is semantically coherent and well-separated, and final object-level segmentation is obtained via unsupervised clustering. This enables fast editing, such as object removal, composition, and style transfer, by directly manipulating the scene's Gaussians. We evaluate TRASE on five dynamic benchmarks, demonstrating state-of-the-art segmentation performance from unseen viewpoints and its effectiveness across various interactive editing tasks. Our project page is available at: https://yunjinli.github.io/project-sadg/