OnlineHMR: Video-based Online World-Grounded Human Mesh Recovery

TL;DR

OnlineHMR introduces a fully online human mesh recovery framework from monocular videos, enabling real-time, world-grounded 3D human motion reconstruction suitable for interactive applications like AR/VR.

Contribution

It presents a novel online architecture with causal inference, sliding-window learning, and incremental SLAM for real-time, world-grounded human mesh recovery from videos.

Findings

Achieves comparable accuracy to chunk-based methods on EMDB benchmark.

Supports real-time processing with high dynamic motion videos.

Maintains temporal consistency and physical plausibility in online mode.

Abstract

Human mesh recovery (HMR) models 3D human body from monocular videos, with recent works extending it to world-coordinate human trajectory and motion reconstruction. However, most existing methods remain offline, relying on future frames or global optimization, which limits their applicability in interactive feedback and perception-action loop scenarios such as AR/VR and telepresence. To address this, we propose OnlineHMR, a fully online framework that jointly satisfies four essential criteria of online processing, including system-level causality, faithfulness, temporal consistency, and efficiency. Built upon a two-branch architecture, OnlineHMR enables streaming inference via a causal key-value cache design and a curated sliding-window learning strategy. Meanwhile, a human-centric incremental SLAM provides online world-grounded alignment under physically plausible trajectory correction. Experimental results show that our method achieves performance comparable to existing chunk-based approaches on the standard EMDB benchmark and highly dynamic custom videos, while uniquely supporting online processing. Page and code are available at https://tsukasane.github.io/Video-OnlineHMR/.