On Coordinate Decoding for Keypoint Estimation Tasks

TL;DR

This paper revisits 2D and 3D keypoint estimation using heatmap representations, emphasizing the importance of encoding and decoding strategies for accurate coordinate prediction and unbiased training.

Contribution

It demonstrates that a distribution-aware decoding method and continuous heatmap reconstruction improve keypoint estimation accuracy and training bias.

Findings

Distribution-aware decoding outperforms standard techniques.

Continuous heatmap reconstruction leads to unbiased training.

Reproduces and extends DARK's findings on heatmap encoding and decoding.

Abstract

A series of 2D (and 3D) keypoint estimation tasks are built upon heatmap coordinate representation, i.e. a probability map that allows for learnable and spatially aware encoding and decoding of keypoint coordinates on grids, even allowing for sub-pixel coordinate accuracy. In this report, we aim to reproduce the findings of DARK that investigated the 2D heatmap representation by highlighting the importance of the encoding of the ground truth heatmap and the decoding of the predicted heatmap to keypoint coordinates. The authors claim that a) a more principled distribution-aware coordinate decoding method overcomes the limitations of the standard techniques widely used in the literature, and b), that the reconstruction of heatmaps from ground-truth coordinates by generating accurate and continuous heatmap distributions lead to unbiased model training, contrary to the standard coordinate encoding process that quantizes the keypoint coordinates on the resolution of the input image grid.