Poseur: Direct Human Pose Regression with Transformers

TL;DR

Poseur introduces a novel Transformer-based regression method for 2D human pose estimation from single images, outperforming heatmap-based approaches by directly predicting keypoints with an attention mechanism.

Contribution

This work is the first to successfully apply a Transformer-based regression approach to 2D human pose estimation, eliminating the need for heatmaps and improving accuracy.

Findings

Outperforms state-of-the-art regression methods.

Favors heatmap-based methods on key benchmarks.

Demonstrates effectiveness of attention in pose regression.

Abstract

We propose a direct, regression-based approach to 2D human pose estimation from single images. We formulate the problem as a sequence prediction task, which we solve using a Transformer network. This network directly learns a regression mapping from images to the keypoint coordinates, without resorting to intermediate representations such as heatmaps. This approach avoids much of the complexity associated with heatmap-based approaches. To overcome the feature misalignment issues of previous regression-based methods, we propose an attention mechanism that adaptively attends to the features that are most relevant to the target keypoints, considerably improving the accuracy. Importantly, our framework is end-to-end differentiable, and naturally learns to exploit the dependencies between keypoints. Experiments on MS-COCO and MPII, two predominant pose-estimation datasets, demonstrate that our method significantly improves upon the state-of-the-art in regression-based pose estimation. More notably, ours is the first regression-based approach to perform favorably compared to the best heatmap-based pose estimation methods.