InstDrive: Instance-Aware 3D Gaussian Splatting for Driving Scenes

TL;DR

InstDrive is a novel 3D Gaussian Splatting framework that achieves instance-aware reconstruction and segmentation of dynamic outdoor driving scenes from dashcam videos, enabling better scene understanding and editing.

Contribution

It introduces a new instance-aware 3D Gaussian Splatting method tailored for outdoor driving scenes, using masks and a lightweight codebook for 3D instance segmentation without complex pre-processing.

Findings

Effective 3D instance segmentation in driving scenes

Outperforms existing methods in outdoor dynamic scene reconstruction

First framework to achieve 3D instance segmentation in open-world driving scenes

Abstract

Reconstructing dynamic driving scenes from dashcam videos has attracted increasing attention due to its significance in autonomous driving and scene understanding. While recent advances have made impressive progress, most methods still unify all background elements into a single representation, hindering both instance-level understanding and flexible scene editing. Some approaches attempt to lift 2D segmentation into 3D space, but often rely on pre-processed instance IDs or complex pipelines to map continuous features to discrete identities. Moreover, these methods are typically designed for indoor scenes with rich viewpoints, making them less applicable to outdoor driving scenarios. In this paper, we present InstDrive, an instance-aware 3D Gaussian Splatting framework tailored for the interactive reconstruction of dynamic driving scene. We use masks generated by SAM as pseudo ground-truth to guide 2D feature learning via contrastive loss and pseudo-supervised objectives. At the 3D level, we introduce regularization to implicitly encode instance identities and enforce consistency through a voxel-based loss. A lightweight static codebook further bridges continuous features and discrete identities without requiring data pre-processing or complex optimization. Quantitative and qualitative experiments demonstrate the effectiveness of InstDrive, and to the best of our knowledge, it is the first framework to achieve 3D instance segmentation in dynamic, open-world driving scenes.More visualizations are available at our project page.