Novel Semantic Prompting for Zero-Shot Action Recognition

TL;DR

This paper introduces SP-CLIP, a semantic prompting framework that enhances zero-shot action recognition by enriching textual descriptions at multiple abstraction levels, improving performance without altering the visual model.

Contribution

The paper proposes a novel semantic prompting approach that leverages structured prompts at various abstraction levels, significantly boosting zero-shot action recognition accuracy.

Findings

Semantic prompting improves zero-shot recognition of fine-grained actions.

SP-CLIP achieves state-of-the-art results on standard benchmarks.

The method maintains model efficiency and generalization capabilities.

Abstract

Zero-shot action recognition relies on transferring knowledge from vision-language models to unseen actions using semantic descriptions. While recent methods focus on temporal modeling or architectural adaptations to handle video data, we argue that semantic prompting alone provides a strong and underexplored signal for zero-shot action understanding. We introduce SP-CLIP, a lightweight framework that augments frozen vision-language models with structured semantic prompts describing actions at multiple levels of abstraction, such as intent, motion, and object interaction. Without modifying the visual encoder or learning additional parameters, SP-CLIP aligns video representations with enriched textual semantics through prompt aggregation and consistency scoring. Experiments across standard benchmarks show that semantic prompting substantially improves zero-shot action recognition, particularly for fine-grained and compositional actions, while preserving the efficiency and generalization of pretrained models.