TrajTok: Learning Trajectory Tokens enables better Video Understanding

TL;DR

TrajTok introduces an end-to-end, adaptive video tokenizer that improves efficiency and performance in video understanding tasks by directly producing object trajectories through implicit clustering.

Contribution

It presents a fully integrated, co-trained video tokenizer that adapts token granularity to semantic complexity, enhancing video understanding without external segmentation pipelines.

Findings

TrajViT2 achieves top accuracy in classification and retrieval benchmarks.

TrajTok improves efficiency comparable to token-merging methods.

It enhances long-video reasoning in vision-language models.

Abstract

Tokenization in video models, typically through patchification, generates an excessive and redundant number of tokens. This severely limits video efficiency and scalability. While recent trajectory-based tokenizers offer a promising solution by decoupling video duration from token count, they rely on complex external segmentation and tracking pipelines that are slow and task-agnostic. We propose TrajTok, an end-to-end video tokenizer module that is fully integrated and co-trained with video models for a downstream objective, dynamically adapting its token granularity to semantic complexity, independent of video duration. TrajTok contains a unified segmenter that performs implicit clustering over pixels in both space and time to directly produce object trajectories in a single forward pass. By prioritizing downstream adaptability over pixel-perfect segmentation fidelity, TrajTok is lightweight and efficient, yet empirically improves video understanding performance. With TrajTok, we implement a video CLIP model trained from scratch (TrajViT2). It achieves the best accuracy at scale across both classification and retrieval benchmarks, while maintaining efficiency comparable to the best token-merging methods. TrajTok also proves to be a versatile component beyond its role as a tokenizer. We show that it can be seamlessly integrated as either a probing head for pretrained visual features (TrajAdapter) or an alignment connector in vision-language models (TrajVLM) with especially strong performance in long-video reasoning.