VideoSTF: Stress-Testing Output Repetition in Video Large Language Models

TL;DR

This paper introduces VideoSTF, a framework for systematically measuring and stress-testing output repetition in Video Large Language Models, revealing widespread repetition issues sensitive to temporal perturbations and exposing security vulnerabilities.

Contribution

We propose VideoSTF, the first standardized benchmark for evaluating output repetition in VideoLLMs, including metrics, a diverse testbed, and stress-testing methods.

Findings

Output repetition is widespread in VideoLLMs.

Repetition is highly sensitive to temporal perturbations.

Simple transformations can induce degeneration, exposing security vulnerabilities.

Abstract

Video Large Language Models (VideoLLMs) have recently achieved strong performance in video understanding tasks. However, we identify a previously underexplored generation failure: severe output repetition, where models degenerate into self-reinforcing loops of repeated phrases or sentences. This failure mode is not captured by existing VideoLLM benchmarks, which focus primarily on task accuracy and factual correctness. We introduce VideoSTF, the first framework for systematically measuring and stress-testing output repetition in VideoLLMs. VideoSTF formalizes repetition using three complementary n-gram-based metrics and provides a standardized testbed of 10,000 diverse videos together with a library of controlled temporal transformations. Using VideoSTF, we conduct pervasive testing, temporal stress testing, and adversarial exploitation across 10 advanced VideoLLMs. We find that output repetition is widespread and, critically, highly sensitive to temporal perturbations of video inputs. Moreover, we show that simple temporal transformations can efficiently induce repetitive degeneration in a black-box setting, exposing output repetition as an exploitable security vulnerability. Our results reveal output repetition as a fundamental stability issue in modern VideoLLMs and motivate stability-aware evaluation for video-language systems. Our evaluation code and scripts are available at: https://github.com/yuxincao22/VideoSTF_benchmark.