Trajectory-aligned Space-time Tokens for Few-shot Action Recognition

TL;DR

This paper introduces trajectory-aligned tokens and a Masked Space-time Transformer to improve few-shot action recognition by disentangling motion and appearance, achieving state-of-the-art results with less data.

Contribution

It presents a novel method combining trajectory-aligned tokens with a transformer to enhance few-shot action recognition, emphasizing disentanglement of motion and appearance.

Findings

Achieves state-of-the-art results on multiple datasets.

Reduces data requirements for effective recognition.

Effectively disentangles motion and appearance representations.

Abstract

We propose a simple yet effective approach for few-shot action recognition, emphasizing the disentanglement of motion and appearance representations. By harnessing recent progress in tracking, specifically point trajectories and self-supervised representation learning, we build trajectory-aligned tokens (TATs) that capture motion and appearance information. This approach significantly reduces the data requirements while retaining essential information. To process these representations, we use a Masked Space-time Transformer that effectively learns to aggregate information to facilitate few-shot action recognition. We demonstrate state-of-the-art results on few-shot action recognition across multiple datasets. Our project page is available at https://www.cs.umd.edu/~pulkit/tats