Learning to Estimate External Forces of Human Motion in Video

TL;DR

This paper introduces a transformer-based method to estimate ground reaction forces from video, improving accuracy and generalization over previous LSTM-based approaches, with applications in sports analysis and injury prevention.

Contribution

It is the first to apply transformer architecture to GRF estimation from video and introduces a new loss function for high impact peaks, enhancing prediction accuracy.

Findings

Up to 19% error reduction compared to prior methods

Transformer architecture outperforms LSTM-based models

Pre-training on pose estimation tasks improves GRF prediction

Abstract

Analyzing sports performance or preventing injuries requires capturing ground reaction forces (GRFs) exerted by the human body during certain movements. Standard practice uses physical markers paired with force plates in a controlled environment, but this is marred by high costs, lengthy implementation time, and variance in repeat experiments; hence, we propose GRF inference from video. While recent work has used LSTMs to estimate GRFs from 2D viewpoints, these can be limited in their modeling and representation capacity. First, we propose using a transformer architecture to tackle the GRF from video task, being the first to do so. Then we introduce a new loss to minimize high impact peaks in regressed curves. We also show that pre-training and multi-task learning on 2D-to-3D human pose estimation improves generalization to unseen motions. And pre-training on this different task provides good initial weights when finetuning on smaller (rarer) GRF datasets. We evaluate on LAAS Parkour and a newly collected ForcePose dataset; we show up to 19% decrease in error compared to prior approaches.