ObjectGS: Object-aware Scene Reconstruction and Scene Understanding via Gaussian Splatting

TL;DR

ObjectGS introduces an innovative framework that combines high-fidelity 3D scene reconstruction with semantic understanding by modeling individual objects as local Gaussian anchors, enabling object-level perception and manipulation.

Contribution

It presents a novel object-aware approach that unifies 3D reconstruction and semantic segmentation, improving object-level accuracy and enabling scene editing capabilities.

Findings

Outperforms state-of-the-art on open-vocabulary segmentation

Enables precise object-level reconstruction and editing

Seamlessly integrates with mesh extraction applications

Abstract

3D Gaussian Splatting is renowned for its high-fidelity reconstructions and real-time novel view synthesis, yet its lack of semantic understanding limits object-level perception. In this work, we propose ObjectGS, an object-aware framework that unifies 3D scene reconstruction with semantic understanding. Instead of treating the scene as a unified whole, ObjectGS models individual objects as local anchors that generate neural Gaussians and share object IDs, enabling precise object-level reconstruction. During training, we dynamically grow or prune these anchors and optimize their features, while a one-hot ID encoding with a classification loss enforces clear semantic constraints. We show through extensive experiments that ObjectGS not only outperforms state-of-the-art methods on open-vocabulary and panoptic segmentation tasks, but also integrates seamlessly with applications like mesh extraction and scene editing. Project page: https://ruijiezhu94.github.io/ObjectGS_page