CycliST: A Video Language Model Benchmark for Reasoning on Cyclical State Transitions

TL;DR

CycliST introduces a benchmark dataset to evaluate Video Language Models on their ability to reason about cyclical state transitions, revealing current models' limitations in understanding periodic patterns and temporal dynamics.

Contribution

This paper presents CycliST, a new synthetic video benchmark for testing VLMs on cyclical reasoning, highlighting the gaps in current models' temporal understanding and generalization capabilities.

Findings

Current VLMs struggle with cyclical motion detection.

Models lack robust temporal reasoning about periodic patterns.

No single model outperforms others across all tasks.

Abstract

We present CycliST, a novel benchmark dataset designed to evaluate Video Language Models (VLM) on their ability for textual reasoning over cyclical state transitions. CycliST captures fundamental aspects of real-world processes by generating synthetic, richly structured video sequences featuring periodic patterns in object motion and visual attributes. CycliST employs a tiered evaluation system that progressively increases difficulty through variations in the number of cyclic objects, scene clutter, and lighting conditions, challenging state-of-the-art models on their spatio-temporal cognition. We conduct extensive experiments with current state-of-the-art VLMs, both open-source and proprietary, and reveal their limitations in generalizing to cyclical dynamics such as linear and orbital motion, as well as time-dependent changes in visual attributes like color and scale. Our results demonstrate that present-day VLMs struggle to reliably detect and exploit cyclic patterns, lack a notion of temporal understanding, and are unable to extract quantitative insights from scenes, such as the number of objects in motion, highlighting a significant technical gap that needs to be addressed. More specifically, we find no single model consistently leads in performance: neither size nor architecture correlates strongly with outcomes, and no model succeeds equally well across all tasks. By providing a targeted challenge and a comprehensive evaluation framework, CycliST paves the way for visual reasoning models that surpass the state-of-the-art in understanding periodic patterns.