Few-shot Action Recognition with Captioning Foundation Models

TL;DR

CapFSAR leverages captioning foundation models to automatically generate textual descriptions from videos, enabling effective multimodal few-shot action recognition without manual annotation, and achieves state-of-the-art results.

Contribution

Introduces CapFSAR, a plug-and-play framework that exploits multimodal foundation models for few-shot action recognition without manual text annotation.

Findings

Outperforms existing methods on multiple benchmarks.

Effectively utilizes automatically generated captions for recognition.

Achieves state-of-the-art performance in few-shot settings.

Abstract

Transferring vision-language knowledge from pretrained multimodal foundation models to various downstream tasks is a promising direction. However, most current few-shot action recognition methods are still limited to a single visual modality input due to the high cost of annotating additional textual descriptions. In this paper, we develop an effective plug-and-play framework called CapFSAR to exploit the knowledge of multimodal models without manually annotating text. To be specific, we first utilize a captioning foundation model (i.e., BLIP) to extract visual features and automatically generate associated captions for input videos. Then, we apply a text encoder to the synthetic captions to obtain representative text embeddings. Finally, a visual-text aggregation module based on Transformer is further designed to incorporate cross-modal spatio-temporal complementary information for reliable few-shot matching. In this way, CapFSAR can benefit from powerful multimodal knowledge of pretrained foundation models, yielding more comprehensive classification in the low-shot regime. Extensive experiments on multiple standard few-shot benchmarks demonstrate that the proposed CapFSAR performs favorably against existing methods and achieves state-of-the-art performance. The code will be made publicly available.