VELOCITI: Benchmarking Video-Language Compositional Reasoning with Strict Entailment

TL;DR

VELOCITI introduces a benchmark for evaluating video-language compositional reasoning, focusing on understanding agents and actions across short videos using strict entailment, revealing significant gaps between current models and human performance.

Contribution

The paper presents VELOCITI, a new benchmark with StrictVLE for assessing video-language reasoning, highlighting current model limitations and emphasizing the importance of visual context in compositional understanding.

Findings

Current models achieve less than 50% accuracy on VELOCITI.

Action understanding is weaker than agent recognition.

Negative captions with entities in videos are more challenging.

Abstract

A fundamental aspect of compositional reasoning in a video is associating people and their actions across time. Recent years have seen great progress in general-purpose vision or video models and a move towards long-video understanding. While exciting, we take a step back and ask: are current models good at compositional reasoning on short videos? To this end, we introduce VELOCITI, a benchmark to study Video-LLMs by disentangling and assessing the comprehension of agents, actions, and their associations across multiple events. We adopt the Video-Language Entailment setup and propose StrictVLE that requires correct classification (rather than ranking) of the positive and negative caption. We evaluate several models and observe that even the best, LLaVA-OneVision (44.5%) and Gemini-1.5-Pro (49.3%), are far from human accuracy at 93.0%. Results show that action understanding lags behind agents, and negative captions created using entities appearing in the video perform worse than those obtained from pure text manipulation. We also present challenges with ClassicVLE and multiple-choice (MC) evaluation, strengthening our preference for StrictVLE. Finally, we validate that our benchmark requires visual inputs of multiple frames making it ideal to study video-language compositional reasoning.