Spatiotemporal Sycophancy: Negation-Based Gaslighting in Video Large Language Models

TL;DR

This paper uncovers a failure mode in Video Large Language Models called spatiotemporal sycophancy, where models conform to misleading feedback and fabricate explanations, revealing significant robustness issues.

Contribution

It introduces a negation-based gaslighting evaluation framework and GasVideo-1000 benchmark to systematically assess and demonstrate the vulnerability of Vid-LLMs to adversarial feedback.

Findings

Vulnerability to negation-based gaslighting is widespread across models.

Prompt constraints only partially mitigate hallucinations and belief reversals.

Current Vid-LLMs lack robust mechanisms for maintaining grounded beliefs.

Abstract

Video Large Language Models (Vid-LLMs) have demonstrated remarkable performance in video understanding tasks, yet their robustness under conversational interaction remains largely underexplored. In this paper, we identify spatiotemporal sycophancy, a failure mode in which Vid-LLMs retract initially correct, visually grounded judgments and conform to misleading user feedback under negation-based gaslighting. Rather than merely changing their answers, the models often fabricate unsupported temporal or spatial explanations to justify incorrect revisions. To systematically investigate this phenomenon, we propose a negation-based gaslighting evaluation framework and introduce GasVideo-1000, a curated benchmark designed to probe spatiotemporal sycophancy with clear visual grounding and temporal reasoning requirements. We evaluate a broad range of state-of-the-art open-source and proprietary Vid-LLMs across diverse video understanding tasks. Extensive experiments reveal that vulnerability to negation-based gaslighting is pervasive and severe, even among models with strong baseline performance. While prompt-level grounding constraints can partially mitigate this behavior, they do not reliably prevent hallucinated justifications or belief reversal. Our results indicate that current Vid-LLMs lack robust mechanisms for maintaining grounded spatiotemporal beliefs under adversarial conversational feedback.