SRL-CLIP: Efficient CLIP Video Adaptation via Structured Semantic Role Labels

TL;DR

This paper introduces SRL-CLIP, an efficient method for adapting CLIP to video understanding by using structured semantic role labels and a small dataset, achieving strong zero-shot performance.

Contribution

The authors propose a novel approach using semantic role labels and rule-based captions to adapt CLIP for holistic video understanding with minimal data and training.

Findings

SRL-CLIP achieves comparable or better performance than larger models.

Contrastive finetuning on only 23k video-caption pairs is sufficient.

SRL-CLIP outperforms CLIP on multiple video benchmarks.

Abstract

Adapting CLIP for videos has gained popularity due to its semantic and rich representation. While CLIP is a good starting point, it typically undergoes post-pretraining (contrastive finetuning) on large video narration or caption datasets (e.g. HowTo100M, WebVid2.5M). However, such narrations or captions often lack comprehensive information needed to represent a video holistically. As the learning signal from text is sparse, the visual learning is inefficient and adaptation requires millions of samples to post-pretrain. In this work, we ask: is it possible to efficiently adapt CLIP for general and holistic video understanding? We use videos labeled with structured and dense Semantic Role Labels (SRLs) that capture actions, people or objects, their attributes, adverbs (manner), and location in a structured format representing the entire video in a holistic way. We generate rule-based captions from SRLs and demonstrate that simple contrastive finetuning on a mere 23k video-caption pairs is adequate to learn powerful, transferable representations applicable across a diverse range of video understanding tasks that require varying levels of perceptual granularity. Our adapted CLIP model, SRL-CLIP, exhibits comparable or superior performance on zero-shot text-to-video retrieval compared to state-of-the-art models that possess 4-8x more parameters and are post-pretrained on up to 6000x more data. SRL-CLIP surpasses CLIP on multiple video benchmarks, underscoring the efficient learning and improved representations.