Learning Dynamical Human-Joint Affinity for 3D Pose Estimation in Videos

TL;DR

This paper introduces DG-Net, a dynamic graph convolutional network that adaptively learns spatial and temporal human-joint relations in videos, improving 3D pose estimation accuracy over fixed-structure methods.

Contribution

The paper proposes a novel DG-Net with dynamical spatial/temporal graph convolutions that adaptively identify joint affinities for better 3D pose estimation in videos.

Findings

DG-Net outperforms recent SOTA methods on benchmarks.

It requires fewer input frames and smaller model size.

Effective in reducing depth ambiguity and motion uncertainty.

Abstract

Graph Convolution Network (GCN) has been successfully used for 3D human pose estimation in videos. However, it is often built on the fixed human-joint affinity, according to human skeleton. This may reduce adaptation capacity of GCN to tackle complex spatio-temporal pose variations in videos. To alleviate this problem, we propose a novel Dynamical Graph Network (DG-Net), which can dynamically identify human-joint affinity, and estimate 3D pose by adaptively learning spatial/temporal joint relations from videos. Different from traditional graph convolution, we introduce Dynamical Spatial/Temporal Graph convolution (DSG/DTG) to discover spatial/temporal human-joint affinity for each video exemplar, depending on spatial distance/temporal movement similarity between human joints in this video. Hence, they can effectively understand which joints are spatially closer and/or have consistent motion, for reducing depth ambiguity and/or motion uncertainty when lifting 2D pose to 3D pose. We conduct extensive experiments on three popular benchmarks, e.g., Human3.6M, HumanEva-I, and MPI-INF-3DHP, where DG-Net outperforms a number of recent SOTA approaches with fewer input frames and model size.