ActAlign: Zero-Shot Fine-Grained Video Classification via Language-Guided Sequence Alignment

TL;DR

ActAlign is a zero-shot, training-free method that aligns language-generated sub-action sequences with video frames using DTW, enabling fine-grained video classification without supervision and outperforming larger models.

Contribution

It introduces a novel sequence alignment approach leveraging language priors for zero-shot fine-grained video classification without training.

Findings

Achieves 30.5% accuracy on ActionAtlas benchmark.

Outperforms larger video-language models with fewer parameters.

Demonstrates effectiveness across diverse sports actions.

Abstract

We address the task of zero-shot video classification for extremely fine-grained actions (e.g., Windmill Dunk in basketball), where no video examples or temporal annotations are available for unseen classes. While image-language models (e.g., CLIP, SigLIP) show strong open-set recognition, they lack temporal modeling needed for video understanding. We propose ActAlign, a truly zero-shot, training-free method that formulates video classification as a sequence alignment problem, preserving the generalization strength of pretrained image-language models. For each class, a large language model (LLM) generates an ordered sequence of sub-actions, which we align with video frames using Dynamic Time Warping (DTW) in a shared embedding space. Without any video-text supervision or fine-tuning, ActAlign achieves 30.5% accuracy on ActionAtlas--the most diverse benchmark of fine-grained actions across multiple sports--where human performance is only 61.6%. ActAlign outperforms billion-parameter video-language models while using 8x fewer parameters. Our approach is model-agnostic and domain-general, demonstrating that structured language priors combined with classical alignment methods can unlock the open-set recognition potential of image-language models for fine-grained video understanding.