Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMs

TL;DR

This paper introduces VISE, a comprehensive benchmark for evaluating sycophantic tendencies in Video-LLMs, and explores strategies to mitigate such biases without additional training.

Contribution

It presents the first systematic benchmark for Video-LLMs' sycophancy, analyzing its manifestations and proposing inference-time mitigation methods.

Findings

VISE enables detailed analysis of sycophantic behavior in Video-LLMs.

Two inference-time strategies can reduce sycophantic bias without retraining.

The benchmark covers diverse question formats and visual reasoning tasks.

Abstract

As video large language models (Video-LLMs) become increasingly integrated into real-world applications that demand grounded multimodal reasoning, ensuring their factual consistency and reliability is of critical importance. However, sycophancy, the tendency of these models to align with user input even when it contradicts the visual evidence, undermines their trustworthiness in such contexts. Current sycophancy research has largely overlooked its specific manifestations in the videolanguage domain, resulting in a notable absence of systematic benchmarks and targeted evaluations to understand how Video-LLMs respond under misleading user input. To fill this gap, we propose VISE(Video-LLM Sycophancy Benchmarking and Evaluation), the first benchmark designed to evaluate sycophantic behavior in state-of-the-art Video-LLMs across diverse question formats, prompt biases, and visual reasoning tasks. Specifically, VISEpioneeringly brings linguistic perspectives on sycophancy into the video domain, enabling fine-grained analysis across multiple sycophancy types and interaction patterns. Furthermore, we propose two potential training-free mitigation strategies revealing potential paths for reducing sycophantic bias: (i) enhancing visual grounding through interpretable key-frame selection and (ii) steering model behavior away from sycophancy via targeted, inference-time intervention on its internal neural representations. Our code is available at https://anonymous.4open.science/r/VideoSycophancy-567F.