VISTA: Video Interaction Spatio-Temporal Analysis Benchmark

TL;DR

VISTA is a comprehensive benchmark for evaluating vision-language models on complex, multi-entity, multi-action video understanding, addressing limitations of existing simple-action benchmarks.

Contribution

VISTA introduces a large-scale, interaction-aware diagnostic benchmark with a unified taxonomy for detailed spatio-temporal analysis of VLMs.

Findings

Evaluated 11 state-of-the-art VLMs on VISTA, revealing specific shortcomings.

Decomposed videos into entities, actions, and relations for detailed diagnostics.

Identified spatio-temporal biases in current models.

Abstract

Existing benchmarks for Vision-Language Models (VLMs) primarily evaluate spatio-temporal understanding on simple single-action videos, closed attribute sets and restricted entity types, failing to capture the freeform, multi-action interactions between diverse entities which characterize real-world video understanding. Furthermore, the lack of a systematic framework for analyzing model failures across complementary spatio-temporal axes hinders comprehensive evaluation. To address these gaps, we introduce VISTA, a Video Interaction Spatio-Temporal Analysis benchmark designed for open-set, multi-entity and multi-action spatio-temporal understanding in VLMs. VISTA decomposes videos into interpretable entities, their associated actions, and relational dynamics, enabling multi-axis diagnostics and unified assessment of relational, spatial, and temporal understanding. Our benchmark integrates multiple datasets into a single interaction-aware taxonomy and comprises ~12K curated video-query pairs spanning diverse scenes and complexities. We systematically evaluate 11 state-of-the-art VLMs on VISTA, and break down aggregate performance across our taxonomy to reveal shortcomings and pronounced spatio-temporal biases obscured by traditional metrics. By providing detailed, taxonomy-driven diagnostics on a challenging dataset, VISTA offers a nuanced framework to guide advances in model design, pretraining strategies, and evaluation protocols. Overall, VISTA is the first, large-scale, interaction-aware diagnostic benchmark for spatio-temporal understanding in VLMs.