Enhancing Spatio-Temporal Zero-shot Action Recognition with Language-driven Description Attributes

TL;DR

This paper introduces a novel approach for zero-shot action recognition that uses web-crawled descriptions and a spatio-temporal module to improve semantic understanding and reduce manual annotation effort, achieving state-of-the-art results.

Contribution

It proposes leveraging large-language models to extract description attributes from web data and a spatio-temporal interaction module to enhance video understanding in zero-shot settings.

Findings

Achieved 81.0% accuracy on UCF-101

Achieved 53.1% accuracy on HMDB-51

Achieved 68.9% accuracy on Kinetics-600

Abstract

Vision-Language Models (VLMs) have demonstrated impressive capabilities in zero-shot action recognition by learning to associate video embeddings with class embeddings. However, a significant challenge arises when relying solely on action classes to provide semantic context, particularly due to the presence of multi-semantic words, which can introduce ambiguity in understanding the intended concepts of actions. To address this issue, we propose an innovative approach that harnesses web-crawled descriptions, leveraging a large-language model to extract relevant keywords. This method reduces the need for human annotators and eliminates the laborious manual process of attribute data creation. Additionally, we introduce a spatio-temporal interaction module designed to focus on objects and action units, facilitating alignment between description attributes and video content. In our zero-shot experiments, our model achieves impressive results, attaining accuracies of 81.0%, 53.1%, and 68.9% on UCF-101, HMDB-51, and Kinetics-600, respectively, underscoring the model's adaptability and effectiveness across various downstream tasks.