EAST: Early Action Prediction Sampling Strategy with Token Masking

TL;DR

EAST is an efficient framework for early action prediction that uses randomized training and token masking to improve generalization, scalability, and state-of-the-art performance across multiple datasets.

Contribution

The paper introduces a novel randomized training strategy and token masking procedure that enhance early action prediction models' generalization and scalability.

Findings

EAST achieves state-of-the-art accuracy on NTU60, SSv2, and UCF101 datasets.

Token masking reduces memory usage by half and doubles training speed with minimal accuracy loss.

Joint learning on observed and future representations significantly improves prediction performance.

Abstract

Early action prediction seeks to anticipate an action before it fully unfolds, but limited visual evidence makes this task especially challenging. We introduce EAST, a simple and efficient framework that enables a model to reason about incomplete observations. In our empirical study, we identify key components when training early action prediction models. Our key contribution is a randomized training strategy that samples a time step separating observed and unobserved video frames, enabling a single model to generalize seamlessly across all test-time observation ratios. We further show that joint learning on both observed and future (oracle) representations significantly boosts performance, even allowing an encoder-only model to excel. To improve scalability, we propose a token masking procedure that cuts memory usage in half and accelerates training by 2x with negligible accuracy loss. Combined with a forecasting decoder, EAST sets a new state of the art on NTU60, SSv2, and UCF101, surpassing previous best work by 10.1, 7.7, and 3.9 percentage points, respectively.