Motion Capture is Not the Target Domain: Scaling Synthetic Data for Learning Motion Representations

TL;DR

This paper investigates the effectiveness of synthetic motion data for pretraining human motion models, revealing that large-scale synthetic data can improve transferability but faces domain mismatch challenges limiting its benefits.

Contribution

The study provides a comprehensive analysis of synthetic motion data for pretraining, highlighting the limits of domain transfer and offering insights into how scaling synthetic data affects model generalization.

Findings

Synthetic pretraining improves generalization when combined with real data.

Large-scale synthetic data yields only marginal gains due to domain mismatch.

Domain mismatch remains a key challenge in synthetic to real transfer for motion models.

Abstract

Synthetic data offers a compelling path to scalable pretraining when real-world data is scarce, but models pretrained on synthetic data often fail to transfer reliably to deployment settings. We study this problem in full-body human motion, where large-scale data collection is infeasible but essential for wearable-based Human Activity Recognition (HAR), and where synthetic motion can be generated from motion-capture-derived representations. We pretrain motion time-series models using such synthetic data and evaluate their transfer across diverse downstream HAR tasks. Our results show that synthetic pretraining improves generalisation when mixed with real data or scaled sufficiently. We also demonstrate that large-scale motion-capture pretraining yields only marginal gains due to domain mismatch with wearable signals, clarifying key sim-to-real challenges and the limits and opportunities of synthetic motion data for transferable HAR representations.