ReMA: A Training-Free Plug-and-Play Mixing Augmentation for Video Behavior Recognition

TL;DR

ReMA is a training-free, plug-and-play augmentation method for video behavior recognition that improves robustness by controlled, class-aware mixing of representations, guided by motion-aware masks and distributional alignment.

Contribution

ReMA introduces a novel, training-free augmentation strategy that controls mixing processes to enhance video representation stability and robustness without extra supervision.

Findings

Consistently improves generalization across benchmarks

Enhances robustness to spatiotemporal variations

No additional training or supervision required

Abstract

Video behavior recognition demands stable and discriminative representations under complex spatiotemporal variations. However, prevailing data augmentation strategies for videos remain largely perturbation-driven, often introducing uncontrolled variations that amplify non-discriminative factors, which finally weaken intra-class distributional structure and representation drift with inconsistent gains across temporal scales. To address these problems, we propose Representation-aware Mixing Augmentation (ReMA), a plug-and-play augmentation strategy that formulates mixing as a controlled replacement process to expand representations while preserving class-conditional stability. ReMA integrates two complementary mechanisms. Firstly, the Representation Alignment Mechanism (RAM) performs structured intra-class mixing under distributional alignment constraints, suppressing irrelevant intra-class drift while enhancing statistical reliability. Then, the Dynamic Selection Mechanism (DSM) generates motion-aware spatiotemporal masks to localize perturbations, guiding them away from discrimination-sensitive regions and promoting temporal coherence. By jointly controlling how and where mixing is applied, ReMA improves representation robustness without additional supervision or trainable parameters. Extensive experiments on diverse video behavior benchmarks demonstrate that ReMA consistently enhances generalization and robustness across different spatiotemporal granularities.