Beyond Motion Pattern: An Empirical Study of Physical Forces for Human Motion Understanding

TL;DR

This study demonstrates that incorporating physically inferred forces into human motion understanding models consistently improves performance across various tasks and benchmarks, especially under challenging conditions.

Contribution

The paper systematically evaluates the impact of physical force cues on human motion understanding, revealing their benefits across multiple tasks and datasets.

Findings

Forces improve gait recognition accuracy, e.g., +0.87% on CASIA-B.

Forces enhance action recognition, e.g., +2.00% on Penn Action.

Forces boost video captioning performance, e.g., ROUGE-L score +0.029.

Abstract

Human motion understanding has advanced rapidly through vision-based progress in recognition, tracking, and captioning. However, most existing methods overlook physical cues such as joint actuation forces that are fundamental in biomechanics. This gap motivates our study: if and when do physically inferred forces enhance motion understanding? By incorporating forces into established motion understanding pipelines, we systematically evaluate their impact across baseline models on 3 major tasks: gait recognition, action recognition, and fine-grained video captioning. Across 8 benchmarks, incorporating forces yields consistent performance gains; for example, on CASIA-B, Rank-1 gait recognition accuracy improved from 89.52% to 90.39% (+0.87), with larger gain observed under challenging conditions: +2.7% when wearing a coat and +3.0% at the side view. On Gait3D, performance also increases from 46.0% to 47.3% (+1.3). In action recognition, CTR-GCN achieved +2.00% on Penn Action, while high-exertion classes like punching/slapping improved by +6.96%. Even in video captioning, Qwen2.5-VL's ROUGE-L score rose from 0.310 to 0.339 (+0.029), indicating that physics-inferred forces enhance temporal grounding and semantic richness. These results demonstrate that force cues can substantially complement visual and kinematic features under dynamic, occluded, or appearance-varying conditions.