Noise-in, Bias-out: Balanced and Real-time MoCap Solving

TL;DR

This paper introduces a machine learning approach for real-time, robust marker-based MoCap that handles noisy data and rare poses, using balanced training and uncertainty modeling to improve performance with affordable sensors.

Contribution

The work presents a novel imbalanced regression technique, a unified representation for training data, and an inverse kinematics adaptation to enhance real-time MoCap robustness and accessibility.

Findings

Improved accuracy in challenging poses.

Robustness to noisy and sparse sensor data.

Effective real-time performance with affordable sensors.

Abstract

Real-time optical Motion Capture (MoCap) systems have not benefited from the advances in modern data-driven modeling. In this work we apply machine learning to solve noisy unstructured marker estimates in real-time and deliver robust marker-based MoCap even when using sparse affordable sensors. To achieve this we focus on a number of challenges related to model training, namely the sourcing of training data and their long-tailed distribution. Leveraging representation learning we design a technique for imbalanced regression that requires no additional data or labels and improves the performance of our model in rare and challenging poses. By relying on a unified representation, we show that training such a model is not bound to high-end MoCap training data acquisition, and exploit the advances in marker-less MoCap to acquire the necessary data. Finally, we take a step towards richer and affordable MoCap by adapting a body model-based inverse kinematics solution to account for measurement and inference uncertainty, further improving performance and robustness. Project page: https://moverseai.github.io/noise-tail