A Framework for Human Pose Estimation in Videos

TL;DR

This paper introduces a novel two-stage tree-based optimization framework for human pose estimation in videos that leverages temporal information and models symmetric body parts more effectively, achieving significant performance improvements.

Contribution

The paper proposes an exact, efficient two-stage tree-based optimization method that models symmetric body parts and spatiotemporal constraints without extra computational complexity.

Findings

Achieved dramatically improved performance over state-of-the-art methods.

Effectively models symmetric body parts using abstract body parts.

Provides an exact solution to the pose estimation problem in videos.

Abstract

In this paper, we present a method to estimate a sequence of human poses in unconstrained videos. We aim to demonstrate that by using temporal information, the human pose estimation results can be improved over image based pose estimation methods. In contrast to the commonly employed graph optimization formulation, which is NP-hard and needs approximate solutions, we formulate this problem into a unified two stage tree-based optimization problem for which an efficient and exact solution exists. Although the proposed method finds an exact solution, it does not sacrifice the ability to model the spatial and temporal constraints between body parts in the frames; in fact it models the {\em symmetric} parts better than the existing methods. The proposed method is based on two main ideas: `Abstraction' and `Association' to enforce the intra- and inter-frame body part constraints without inducing extra computational complexity to the polynomial time solution. Using the idea of `Abstraction', a new concept of `abstract body part' is introduced to conceptually combine the symmetric body parts and model them in the tree based body part structure. Using the idea of `Association', the optimal tracklets are generated for each abstract body part, in order to enforce the spatiotemporal constraints between body parts in adjacent frames. A sequence of the best poses is inferred from the abstract body part tracklets through the tree-based optimization. Finally, the poses are refined by limb alignment and refinement schemes. We evaluated the proposed method on three publicly available video based human pose estimation datasets, and obtained dramatically improved performance compared to the state-of-the-art methods.