Punching Bag vs. Punching Person: Motion Transferability in Videos

TL;DR

This paper investigates how well action recognition models transfer high-level motion understanding across different contexts, introducing new datasets and benchmarks to evaluate their generalization capabilities and analyzing factors affecting transferability.

Contribution

The study introduces a motion transferability framework with new synthetic and adapted datasets, providing a benchmark for evaluating and understanding motion transfer in action recognition models.

Findings

Multimodal models struggle more with unknown fine-grained actions.

Synthetic dataset challenges models as real-world datasets do.

Larger models excel with spatial cues but not with temporal reasoning.

Abstract

Action recognition models demonstrate strong generalization, but can they effectively transfer high-level motion concepts across diverse contexts, even within similar distributions? For example, can a model recognize the broad action "punching" when presented with an unseen variation such as "punching person"? To explore this, we introduce a motion transferability framework with three datasets: (1) Syn-TA, a synthetic dataset with 3D object motions; (2) Kinetics400-TA; and (3) Something-Something-v2-TA, both adapted from natural video datasets. We evaluate 13 state-of-the-art models on these benchmarks and observe a significant drop in performance when recognizing high-level actions in novel contexts. Our analysis reveals: 1) Multimodal models struggle more with fine-grained unknown actions than with coarse ones; 2) The bias-free Syn-TA proves as challenging as real-world datasets, with models showing greater performance drops in controlled settings; 3) Larger models improve transferability when spatial cues dominate but struggle with intensive temporal reasoning, while reliance on object and background cues hinders generalization. We further explore how disentangling coarse and fine motions can improve recognition in temporally challenging datasets. We believe this study establishes a crucial benchmark for assessing motion transferability in action recognition. Datasets and relevant code: https://github.com/raiyaan-abdullah/Motion-Transfer.