RecurGS: Interactive Scene Modeling via Discrete-State Recurrent Gaussian Fusion

TL;DR

RecurGS is a novel recurrent fusion framework that incrementally models interactive 3D scenes by detecting object changes, aligning motion, and fusing observations to support continuous scene updates and manipulations.

Contribution

It introduces a recurrent Gaussian scene modeling approach that fuses multiple observations, detects object changes, and supports interactive scene manipulation.

Findings

High-quality scene reconstructions demonstrated on synthetic and real datasets.

Significantly improved update efficiency over existing methods.

Supports object-level manipulation without additional scans.

Abstract

Recent advances in 3D scene representations have enabled high-fidelity novel view synthesis, yet adapting to discrete scene changes and constructing interactive 3D environments remain open challenges in vision and robotics. Existing approaches focus solely on updating a single scene without supporting novel-state synthesis. Others rely on diffusion-based object-background decoupling that works on one state at a time and cannot fuse information across multiple observations. To address these limitations, we introduce RecurGS, a recurrent fusion framework that incrementally integrates discrete Gaussian scene states into a single evolving representation capable of interaction. RecurGS detects object-level changes across consecutive states, aligns their geometric motion using semantic correspondence and Lie-algebra based SE(3) refinement, and performs recurrent updates that preserve historical structures through replay supervision. A voxelized, visibility-aware fusion module selectively incorporates newly observed regions while keeping stable areas fixed, mitigating catastrophic forgetting and enabling efficient long-horizon updates. RecurGS supports object-level manipulation, synthesizes novel scene states without requiring additional scans, and maintains photorealistic fidelity across evolving environments. Extensive experiments across synthetic and real-world datasets demonstrate that our framework delivers high-quality reconstructions with substantially improved update efficiency, providing a scalable step toward continuously interactive Gaussian worlds.