HierVL: Learning Hierarchical Video-Language Embeddings

TL;DR

HierVL introduces a hierarchical approach to video-language embeddings that captures both short-term and long-term associations, improving understanding of video content and context for various tasks.

Contribution

The paper presents HierVL, a novel hierarchical contrastive training method that aligns text and visual data at multiple levels, enhancing long-term video understanding.

Findings

Outperforms single-level embeddings in short-term video tasks

Achieves state-of-the-art results on long-term video modeling benchmarks

Successfully transfers to multiple downstream tasks in zero-shot and fine-tuned settings

Abstract

Video-language embeddings are a promising avenue for injecting semantics into visual representations, but existing methods capture only short-term associations between seconds-long video clips and their accompanying text. We propose HierVL, a novel hierarchical video-language embedding that simultaneously accounts for both long-term and short-term associations. As training data, we take videos accompanied by timestamped text descriptions of human actions, together with a high-level text summary of the activity throughout the long video (as are available in Ego4D). We introduce a hierarchical contrastive training objective that encourages text-visual alignment at both the clip level and video level. While the clip-level constraints use the step-by-step descriptions to capture what is happening in that instant, the video-level constraints use the summary text to capture why it is happening, i.e., the broader context for the activity and the intent of the actor. Our hierarchical scheme yields a clip representation that outperforms its single-level counterpart as well as a long-term video representation that achieves SotA results on tasks requiring long-term video modeling. HierVL successfully transfers to multiple challenging downstream tasks (in EPIC-KITCHENS-100, Charades-Ego, HowTo100M) in both zero-shot and fine-tuned settings.