Building a Multi-modal Spatiotemporal Expert for Zero-shot Action Recognition with CLIP

TL;DR

This paper introduces STDD, a CLIP-based framework that enhances zero-shot action recognition by capturing multi-modal spatiotemporal dynamics through innovative attention and text augmentation, outperforming existing methods on key benchmarks.

Contribution

The work proposes a novel spatiotemporal dynamic framework with space-time cross attention and an action semantic knowledge graph for improved zero-shot action recognition.

Findings

Outperforms state-of-the-art on Kinetics-600, UCF101, HMDB51

Effectively captures spatiotemporal dynamics without extra parameters

Enhances generalization through aligned video and text representations

Abstract

Zero-shot action recognition (ZSAR) requires collaborative multi-modal spatiotemporal understanding. However, finetuning CLIP directly for ZSAR yields suboptimal performance, given its inherent constraints in capturing essential temporal dynamics from both vision and text perspectives, especially when encountering novel actions with fine-grained spatiotemporal discrepancies. In this work, we propose Spatiotemporal Dynamic Duo (STDD), a novel CLIP-based framework to comprehend multi-modal spatiotemporal dynamics synergistically. For the vision side, we propose an efficient Space-time Cross Attention, which captures spatiotemporal dynamics flexibly with simple yet effective operations applied before and after spatial attention, without adding additional parameters or increasing computational complexity. For the semantic side, we conduct spatiotemporal text augmentation by comprehensively constructing an Action Semantic Knowledge Graph (ASKG) to derive nuanced text prompts. The ASKG elaborates on static and dynamic concepts and their interrelations, based on the idea of decomposing actions into spatial appearances and temporal motions. During the training phase, the frame-level video representations are meticulously aligned with prompt-level nuanced text representations, which are concurrently regulated by the video representations from the frozen CLIP to enhance generalizability. Extensive experiments validate the effectiveness of our approach, which consistently surpasses state-of-the-art approaches on popular video benchmarks (i.e., Kinetics-600, UCF101, and HMDB51) under challenging ZSAR settings.