Motion Capture from Pan-Tilt Cameras with Unknown Orientation

TL;DR

This paper introduces a novel method for 3D motion capture of athletes using freely rotating pan-tilt cameras, combining pose estimation, camera orientation recovery, and bundle adjustment, enabling accurate global pose reconstruction without markers.

Contribution

It presents a new approach that estimates camera orientations and athlete poses simultaneously, allowing large-volume capture with rotating cameras and using a new skiing dataset.

Findings

Accurate 3D athlete pose estimation from images alone.

Effective joint optimization of camera rotation and athlete pose.

Creation of a new annotated skiing dataset for pose estimation.

Abstract

In sports, such as alpine skiing, coaches would like to know the speed and various biomechanical variables of their athletes and competitors. Existing methods use either body-worn sensors, which are cumbersome to setup, or manual image annotation, which is time consuming. We propose a method for estimating an athlete's global 3D position and articulated pose using multiple cameras. By contrast to classical markerless motion capture solutions, we allow cameras to rotate freely so that large capture volumes can be covered. In a first step, tight crops around the skier are predicted and fed to a 2D pose estimator network. The 3D pose is then reconstructed using a bundle adjustment method. Key to our solution is the rotation estimation of Pan-Tilt cameras in a joint optimization with the athlete pose and conditioning on relative background motion computed with feature tracking. Furthermore, we created a new alpine skiing dataset and annotated it with 2D pose labels, to overcome shortcomings of existing ones. Our method estimates accurate global 3D poses from images only and provides coaches with an automatic and fast tool for measuring and improving an athlete's performance.