Video-LevelGauge: Investigating Contextual Positional Bias in Large Video Language Models

TL;DR

Video-LevelGauge is a benchmark designed to systematically evaluate and analyze positional bias in large video language models, revealing significant biases in open-source models and robustness in commercial ones.

Contribution

The paper introduces Video-LevelGauge, a novel benchmark with standardized probes and analysis methods to assess positional bias in LVLMs, filling a gap in nuanced performance evaluation.

Findings

Many open-source LVLMs exhibit head or neighbor-content biases.

Commercial models like Gemini2.5-Pro demonstrate consistent performance across videos.

Analysis of context length and model scale offers insights for bias mitigation.

Abstract

Large video language models (LVLMs) have made notable progress in video understanding, spurring the development of corresponding evaluation benchmarks. However, existing benchmarks generally assess overall performance across entire video sequences, overlooking nuanced behaviors such as contextual positional bias, a critical yet under-explored aspect of LVLM performance. We present Video-LevelGauge, a dedicated benchmark designed to systematically assess positional bias in LVLMs. We employ standardized probes and customized contextual setups, allowing flexible control over context length, probe position, and contextual types to simulate diverse real-world scenarios. In addition, we introduce a comprehensive analysis method that combines statistical measures with morphological pattern recognition to characterize bias. Our benchmark comprises 438 manually curated videos spanning multiple types, yielding 1,177 high-quality multiple-choice questions and 120 open-ended questions, validated for their effectiveness in exposing positional bias. Based on these, we evaluate 27 state-of-the-art LVLMs, including both commercial and open-source models. Our findings reveal significant positional biases in many leading open-source models, typically exhibiting head or neighbor-content preferences. In contrast, commercial models such as Gemini2.5-Pro show impressive, consistent performance across entire video sequences. Further analyses on context length, context variation, and model scale provide actionable insights for mitigating bias and guiding model enhancement . https://github.com/Cola-any/Video-LevelGauge