PoseBench: Benchmarking the Robustness of Pose Estimation Models under Corruptions

TL;DR

PoseBench provides a comprehensive evaluation of 60 pose estimation models' robustness against real-world corruptions, revealing vulnerabilities and guiding improvements for safer, more reliable applications.

Contribution

This work introduces PoseBench, the first extensive benchmark assessing the robustness of pose estimation models under diverse real-world corruptions.

Findings

State-of-the-art models are vulnerable to common corruptions.

Models behave differently on human vs. animal pose tasks.

Design choices like input resolution and data augmentation affect robustness.

Abstract

Pose estimation aims to accurately identify anatomical keypoints in humans and animals using monocular images, which is crucial for various applications such as human-machine interaction, embodied AI, and autonomous driving. While current models show promising results, they are typically trained and tested on clean data, potentially overlooking the corruption during real-world deployment and thus posing safety risks in practical scenarios. To address this issue, we introduce PoseBench, a comprehensive benchmark designed to evaluate the robustness of pose estimation models against real-world corruption. We evaluated 60 representative models, including top-down, bottom-up, heatmap-based, regression-based, and classification-based methods, across three datasets for human and animal pose estimation. Our evaluation involves 10 types of corruption in four categories: 1) blur and noise, 2) compression and color loss, 3) severe lighting, and 4) masks. Our findings reveal that state-of-the-art models are vulnerable to common real-world corruptions and exhibit distinct behaviors when tackling human and animal pose estimation tasks. To improve model robustness, we delve into various design considerations, including input resolution, pre-training datasets, backbone capacity, post-processing, and data augmentations. We hope that our benchmark will serve as a foundation for advancing research in robust pose estimation. The benchmark and source code will be released at https://xymsh.github.io/PoseBench