Towards an Effective Action-Region Tracking Framework for Fine-grained Video Action Recognition

TL;DR

This paper introduces the Action-Region Tracking framework for fine-grained video action recognition, effectively capturing subtle local details over time using a novel query-response mechanism and semantic constraints.

Contribution

The work proposes a new framework that leverages semantic queries and contrastive learning to improve fine-grained action recognition by tracking local region dynamics.

Findings

Outperforms previous state-of-the-art methods on benchmark datasets.

Effectively captures subtle local action details over time.

Utilizes a novel semantic query mechanism for region response detection.

Abstract

Fine-grained action recognition (FGAR) aims to identify subtle and distinctive differences among fine-grained action categories. However, current recognition methods often capture coarse-grained motion patterns but struggle to identify subtle details in local regions evolving over time. In this work, we introduce the Action-Region Tracking (ART) framework, a novel solution leveraging a query-response mechanism to discover and track the dynamics of distinctive local details, enabling effective distinction of similar actions. Specifically, we propose a region-specific semantic activation module that employs discriminative and text-constrained semantics as queries to capture the most action-related region responses in each video frame, facilitating interaction among spatial and temporal dimensions with corresponding video features. The captured region responses are organized into action tracklets, which characterize region-based action dynamics by linking related responses across video frames in a coherent sequence. The text-constrained queries encode nuanced semantic representations derived from textual descriptions of action labels extracted by language branches within Visual Language Models (VLMs). To optimize the action tracklets, we design a multi-level tracklet contrastive constraint among region responses at spatial and temporal levels, enabling effective discrimination within each frame and correlation between adjacent frames. Additionally, a task-specific fine-tuning mechanism refines textual semantics such that semantic representations encoded by VLMs are preserved while optimized for task preferences. Comprehensive experiments on widely used action recognition benchmarks demonstrate the superiority to previous state-of-the-art baselines.