Storyboard guided Alignment for Fine-grained Video Action Recognition

TL;DR

This paper introduces a multi-granularity framework for fine-grained video action recognition that leverages storyboarding-inspired fine-grained descriptions and key frame aggregation to improve video-text alignment accuracy.

Contribution

It proposes a novel approach combining global semantics with fine-grained descriptions generated by large language models for better video-text matching.

Findings

Superior performance in supervised, few-shot, and zero-shot settings

Effective identification of key frames for embedding aggregation

Enhanced alignment accuracy in fine-grained action recognition

Abstract

Fine-grained video action recognition can be conceptualized as a video-text matching problem. Previous approaches often rely on global video semantics to consolidate video embeddings, which can lead to misalignment in video-text pairs due to a lack of understanding of action semantics at an atomic granularity level. To tackle this challenge, we propose a multi-granularity framework based on two observations: (i) videos with different global semantics may share similar atomic actions or appearances, and (ii) atomic actions within a video can be momentary, slow, or even non-directly related to the global video semantics. Inspired by the concept of storyboarding, which disassembles a script into individual shots, we enhance global video semantics by generating fine-grained descriptions using a pre-trained large language model. These detailed descriptions capture common atomic actions depicted in videos. A filtering metric is proposed to select the descriptions that correspond to the atomic actions present in both the videos and the descriptions. By employing global semantics and fine-grained descriptions, we can identify key frames in videos and utilize them to aggregate embeddings, thereby making the embedding more accurate. Extensive experiments on various video action recognition datasets demonstrate superior performance of our proposed method in supervised, few-shot, and zero-shot settings.