OnlineX: Unified Online 3D Reconstruction and Understanding with Active-to-Stable State Evolution

TL;DR

OnlineX is a real-time online 3D reconstruction framework that effectively balances local detail capture and global structure preservation by decoupling active and stable memory states, enabling continuous scene understanding from streaming images.

Contribution

It introduces a novel decoupled active-to-stable state evolution paradigm for online 3D reconstruction, integrating visual and language fields with Gaussian fusion for improved accuracy.

Findings

Outperforms prior methods in novel view synthesis

Achieves robust semantic understanding across sequences

Operates with real-time inference speed

Abstract

Recent advances in generalizable 3D Gaussian Splatting (3DGS) have enabled rapid 3D scene reconstruction within seconds, eliminating the need for per-scene optimization. However, existing methods primarily follow an offline reconstruction paradigm, lacking the capacity for continuous reconstruction, which limits their applicability to online scenarios such as robotics and VR/AR. In this paper, we introduce OnlineX, a feed-forward framework that reconstructs both 3D visual appearance and language fields in an online manner using only streaming images. A key challenge in online formulation is the cumulative drift issue, which is rooted in the fundamental conflict between two opposing roles of the memory state: an active role that constantly refreshes to capture high-frequency local geometry, and a stable role that conservatively accumulates and preserves the long-term global structure. To address this, we introduce a decoupled active-to-stable state evolution paradigm. Our framework decouples the memory state into a dedicated active state and a persistent stable state, and then cohesively fuses the information from the former into the latter to achieve both fidelity and stability. Moreover, we jointly model visual appearance and language fields and incorporate an implicit Gaussian fusion module to enhance reconstruction quality. Experiments on mainstream datasets demonstrate that our method consistently outperforms prior work in novel view synthesis and semantic understanding, showcasing robust performance across input sequences of varying lengths with real-time inference speed.