PAS3R: Pose-Adaptive Streaming 3D Reconstruction for Long Video Sequences

TL;DR

PAS3R is a novel pose-adaptive streaming framework for 3D scene reconstruction from long videos, dynamically weighting frames based on motion and scene change to improve accuracy and stability.

Contribution

It introduces a motion-aware update mechanism and trajectory-consistent training objectives for improved long-term 3D reconstruction stability.

Findings

Significantly improves trajectory accuracy in long sequences

Enhances depth estimation and point cloud quality

Maintains competitive performance on shorter sequences

Abstract

Online monocular 3D reconstruction enables dense scene recovery from streaming video but remains fundamentally limited by the stability-adaptation dilemma: the reconstruction model must rapidly incorporate novel viewpoints while preserving previously accumulated scene structure. Existing streaming approaches rely on uniform or attention-based update mechanisms that often fail to account for abrupt viewpoint transitions, leading to trajectory drift and geometric inconsistencies over long sequences. We introduce PAS3R, a pose-adaptive streaming reconstruction framework that dynamically modulates state updates according to camera motion and scene structure. Our key insight is that frames contributing significant geometric novelty should exert stronger influence on the reconstruction state, while frames with minor viewpoint variation should prioritize preserving historical context. PAS3R operationalizes this principle through a motion-aware update mechanism that jointly leverages inter-frame pose variation and image frequency cues to estimate frame importance. To further stabilize long-horizon reconstruction, we introduce trajectory-consistent training objectives that incorporate relative pose constraints and acceleration regularization. A lightweight online stabilization module further suppresses high-frequency trajectory jitter and geometric artifacts without increasing memory consumption. Extensive experiments across multiple benchmarks demonstrate that PAS3R significantly improves trajectory accuracy, depth estimation, and point cloud reconstruction quality in long video sequences while maintaining competitive performance on shorter sequences.