TRAM: Global Trajectory and Motion of 3D Humans from in-the-wild Videos

TL;DR

TRAM is a novel two-stage approach that combines SLAM and transformer-based regression to accurately reconstruct 3D human trajectories and motions from in-the-wild videos, overcoming challenges posed by dynamic scenes.

Contribution

The paper introduces TRAM, which robustifies SLAM for dynamic scenes and employs a transformer model for precise 3D human motion estimation in world coordinates.

Findings

Reduces global motion errors significantly compared to prior methods

Successfully recovers camera and human motion in complex, dynamic scenes

Achieves accurate 3D human reconstruction in real-world videos

Abstract

We propose TRAM, a two-stage method to reconstruct a human's global trajectory and motion from in-the-wild videos. TRAM robustifies SLAM to recover the camera motion in the presence of dynamic humans and uses the scene background to derive the motion scale. Using the recovered camera as a metric-scale reference frame, we introduce a video transformer model (VIMO) to regress the kinematic body motion of a human. By composing the two motions, we achieve accurate recovery of 3D humans in the world space, reducing global motion errors by a large margin from prior work. https://yufu-wang.github.io/tram4d/