Live Stream Temporally Embedded 3D Human Body Pose and Shape Estimation

TL;DR

This paper introduces TePose, a real-time method for 3D human pose and shape estimation from live stream videos, utilizing temporal information and adversarial training to enhance accuracy and consistency.

Contribution

The paper proposes a novel temporally embedded approach with a multi-scale spatio-temporal graph convolutional network and a sequential data loading strategy for live streams.

Findings

TePose achieves state-of-the-art performance on human pose benchmarks.

The method improves temporal consistency in live stream videos.

Extensive experiments validate the effectiveness of each module.

Abstract

3D Human body pose and shape estimation within a temporal sequence can be quite critical for understanding human behavior. Despite the significant progress in human pose estimation in the recent years, which are often based on single images or videos, human motion estimation on live stream videos is still a rarely-touched area considering its special requirements for real-time output and temporal consistency. To address this problem, we present a temporally embedded 3D human body pose and shape estimation (TePose) method to improve the accuracy and temporal consistency of pose estimation in live stream videos. TePose uses previous predictions as a bridge to feedback the error for better estimation in the current frame and to learn the correspondence between data frames and predictions in the history. A multi-scale spatio-temporal graph convolutional network is presented as the motion discriminator for adversarial training using datasets without any 3D labeling. We propose a sequential data loading strategy to meet the special start-to-end data processing requirement of live stream. We demonstrate the importance of each proposed module with extensive experiments. The results show the effectiveness of TePose on widely-used human pose benchmarks with state-of-the-art performance.