Are Pose Estimators Ready for the Open World? STAGE: A GenAI Toolkit for Auditing 3D Human Pose Estimators

TL;DR

This paper introduces STAGE, a GenAI toolkit for creating controlled image datasets to evaluate the robustness of 3D human pose estimators against various real-world factors, revealing significant performance vulnerabilities.

Contribution

We develop the first GenAI image creator with precise 3D pose control and propose a novel evaluation strategy for isolating effects of single attributes on pose estimator robustness.

Findings

Natural variations significantly degrade pose estimation accuracy.

Pose estimators show sensitivity to gender, ethnicity, and weather changes.

STAGE provides a new benchmark for robustness evaluation.

Abstract

For safety-critical applications, it is crucial to audit 3D human pose estimators before deployment. Will the system break down if the weather or the clothing changes? Is it robust regarding gender and age? To answer these questions and more, we need controlled studies with images that differ in a single attribute, but real benchmarks cannot provide such pairs. We thus present STAGE, a GenAI data toolkit for auditing 3D human pose estimators. For STAGE, we develop the first GenAI image creator with accurate 3D pose control and propose a novel evaluation strategy to isolate and quantify the effects of single factors such as gender, ethnicity, age, clothing, location, and weather. Enabled by STAGE, we generate a series of benchmarks to audit, for the first time, the sensitivity of popular pose estimators towards such factors. Our results show that natural variations can severely degrade pose estimator performance, raising doubts about their readiness for open-world deployment. We aim to highlight these robustness issues and establish STAGE as a benchmark to quantify them.