Trokens: Semantic-Aware Relational Trajectory Tokens for Few-Shot Action Recognition

TL;DR

Trokens introduces semantic-aware relational trajectory tokens that adaptively select and model motion patterns for improved few-shot action recognition across multiple benchmarks.

Contribution

The paper proposes a novel semantic-aware sampling and motion modeling framework that enhances trajectory-based features for few-shot action recognition.

Findings

Achieves state-of-the-art results on six benchmarks.

Effectively models intra- and inter-trajectory motion patterns.

Improves recognition accuracy by combining semantic and motion features.

Abstract

Video understanding requires effective modeling of both motion and appearance information, particularly for few-shot action recognition. While recent advances in point tracking have been shown to improve few-shot action recognition, two fundamental challenges persist: selecting informative points to track and effectively modeling their motion patterns. We present Trokens, a novel approach that transforms trajectory points into semantic-aware relational tokens for action recognition. First, we introduce a semantic-aware sampling strategy to adaptively distribute tracking points based on object scale and semantic relevance. Second, we develop a motion modeling framework that captures both intra-trajectory dynamics through the Histogram of Oriented Displacements (HoD) and inter-trajectory relationships to model complex action patterns. Our approach effectively combines these trajectory tokens with semantic features to enhance appearance features with motion information, achieving state-of-the-art performance across six diverse few-shot action recognition benchmarks: Something-Something-V2 (both full and small splits), Kinetics, UCF101, HMDB51, and FineGym. For project page see https://trokens-iccv25.github.io