SteerPose: Simultaneous Extrinsic Camera Calibration and Matching from Articulation

TL;DR

SteerPose is a neural network that simultaneously calibrates multi-camera systems and matches 2D poses across views by learning pose rotations, enabling 3D pose reconstruction of animals and humans in diverse settings.

Contribution

We introduce SteerPose, a unified framework that combines pose rotation, correspondence, and calibration, inspired by human cognitive abilities, with a novel geometric consistency loss.

Findings

Effective on diverse in-the-wild datasets

Robust in multi-camera setups

Enables 3D pose reconstruction of novel animals

Abstract

Can freely moving humans or animals themselves serve as calibration targets for multi-camera systems while simultaneously estimating their correspondences across views? We humans can solve this problem by mentally rotating the observed 2D poses and aligning them with those in the target views. Inspired by this cognitive ability, we propose SteerPose, a neural network that performs this rotation of 2D poses into another view. By integrating differentiable matching, SteerPose simultaneously performs extrinsic camera calibration and correspondence search within a single unified framework. We also introduce a novel geometric consistency loss that explicitly ensures that the estimated rotation and correspondences result in a valid translation estimation. Experimental results on diverse in-the-wild datasets of humans and animals validate the effectiveness and robustness of the proposed method. Furthermore, we demonstrate that our method can reconstruct the 3D poses of novel animals in multi-camera setups by leveraging off-the-shelf 2D pose estimators and our class-agnostic model.