Reinforcement Learning meets Masked Video Modeling : Trajectory-Guided Adaptive Token Selection

TL;DR

This paper introduces TATS, a trajectory-aware token sampler for masked video modeling, which improves action recognition performance by adaptively selecting motion-centric tokens during pre-training.

Contribution

The work proposes a novel TATS method that models token motion dynamics and integrates with MAE, enabling efficient, adaptive, and motion-focused token selection in video pre-training.

Findings

TATS enables aggressive masking without performance loss.

The approach improves transferability and generalization across benchmarks.

It achieves state-of-the-art results on multiple video action recognition datasets.

Abstract

Masked video modeling~(MVM) has emerged as a highly effective pre-training strategy for visual foundation models, whereby the model reconstructs masked spatiotemporal tokens using information from visible tokens. However, a key challenge in such approaches lies in selecting an appropriate masking strategy. Previous studies have explored predefined masking techniques, including random and tube-based masking, as well as approaches that leverage key motion priors, optical flow and semantic cues from externally pre-trained models. In this work, we introduce a novel and generalizable Trajectory-Aware Adaptive Token Sampler (TATS), which models the motion dynamics of tokens and can be seamlessly integrated into the masked autoencoder (MAE) framework to select motion-centric tokens in videos. Additionally, we propose a unified training strategy that enables joint optimization of both MAE and TATS from scratch using Proximal Policy Optimization (PPO). We show that our model allows for aggressive masking without compromising performance on the downstream task of action recognition while also ensuring that the pre-training remains memory efficient. Extensive experiments of the proposed approach across four benchmarks, including Something-Something v2, Kinetics-400, UCF101, and HMDB51, demonstrate the effectiveness, transferability, generalization, and efficiency of our work compared to other state-of-the-art methods.