TRACE: 5D Temporal Regression of Avatars with Dynamic Cameras in 3D Environments

TL;DR

TRACE is a novel end-to-end method that jointly estimates and tracks 3D human poses in global coordinates from moving cameras, addressing challenges of entangled human and camera motion.

Contribution

It introduces a 5D representation and new maps for trajectory reasoning, enabling the first one-stage global 3D human tracking and pose estimation from dynamic cameras.

Findings

Achieves state-of-the-art tracking performance

Outperforms previous methods in 3D pose estimation

Handles long occlusions effectively

Abstract

Although the estimation of 3D human pose and shape (HPS) is rapidly progressing, current methods still cannot reliably estimate moving humans in global coordinates, which is critical for many applications. This is particularly challenging when the camera is also moving, entangling human and camera motion. To address these issues, we adopt a novel 5D representation (space, time, and identity) that enables end-to-end reasoning about people in scenes. Our method, called TRACE, introduces several novel architectural components. Most importantly, it uses two new "maps" to reason about the 3D trajectory of people over time in camera, and world, coordinates. An additional memory unit enables persistent tracking of people even during long occlusions. TRACE is the first one-stage method to jointly recover and track 3D humans in global coordinates from dynamic cameras. By training it end-to-end, and using full image information, TRACE achieves state-of-the-art performance on tracking and HPS benchmarks. The code and dataset are released for research purposes.