Revisiting the "Video" in Video-Language Understanding

TL;DR

This paper introduces the atemporal probe (ATP), a new model for video-language analysis that helps evaluate and improve understanding of temporal aspects in videos, revealing current benchmarks often overlook temporal complexity.

Contribution

The paper proposes ATP as a novel model to bound baseline accuracy and enhance dataset and model design for better temporal understanding in video-language tasks.

Findings

Understanding event temporality is often not required for strong performance.

ATP can improve dataset analysis by identifying temporally challenging data.

Integrating ATP into models enhances efficiency and accuracy.

Abstract

What makes a video task uniquely suited for videos, beyond what can be understood from a single image? Building on recent progress in self-supervised image-language models, we revisit this question in the context of video and language tasks. We propose the atemporal probe (ATP), a new model for video-language analysis which provides a stronger bound on the baseline accuracy of multimodal models constrained by image-level understanding. By applying this model to standard discriminative video and language tasks, such as video question answering and text-to-video retrieval, we characterize the limitations and potential of current video-language benchmarks. We find that understanding of event temporality is often not necessary to achieve strong or state-of-the-art performance, even compared with recent large-scale video-language models and in contexts intended to benchmark deeper video-level understanding. We also demonstrate how ATP can improve both video-language dataset and model design. We describe a technique for leveraging ATP to better disentangle dataset subsets with a higher concentration of temporally challenging data, improving benchmarking efficacy for causal and temporal understanding. Further, we show that effectively integrating ATP into full video-level temporal models can improve efficiency and state-of-the-art accuracy.