Tracking People with 3D Representations

TL;DR

This paper introduces a 3D representation-based method for multi-person tracking in videos, demonstrating improved accuracy over traditional 2D approaches by using detailed 3D geometry and appearance features.

Contribution

The paper presents HMAR, a novel method that extracts 3D human meshes and textures for robust tracking, and demonstrates its effectiveness with state-of-the-art results.

Findings

3D representations outperform 2D in tracking accuracy

State-of-the-art performance on Posetrack, MuPoTs, AVA datasets

Robustness to viewpoint and pose variations

Abstract

We present a novel approach for tracking multiple people in video. Unlike past approaches which employ 2D representations, we focus on using 3D representations of people, located in three-dimensional space. To this end, we develop a method, Human Mesh and Appearance Recovery (HMAR) which in addition to extracting the 3D geometry of the person as a SMPL mesh, also extracts appearance as a texture map on the triangles of the mesh. This serves as a 3D representation for appearance that is robust to viewpoint and pose changes. Given a video clip, we first detect bounding boxes corresponding to people, and for each one, we extract 3D appearance, pose, and location information using HMAR. These embedding vectors are then sent to a transformer, which performs spatio-temporal aggregation of the representations over the duration of the sequence. The similarity of the resulting representations is used to solve for associations that assigns each person to a tracklet. We evaluate our approach on the Posetrack, MuPoTs and AVA datasets. We find that 3D representations are more effective than 2D representations for tracking in these settings, and we obtain state-of-the-art performance. Code and results are available at: https://brjathu.github.io/T3DP.