Have the VLMs Lost Confidence? A Study of Sycophancy in VLMs

TL;DR

This paper investigates sycophancy in visual language models (VLMs), introduces a benchmark to evaluate it, and proposes methods to mitigate this issue, revealing that improving image attention at higher layers reduces sycophantic behavior.

Contribution

The study extends sycophancy analysis from LLMs to VLMs, introduces the MM-SY benchmark, and proposes training methods to effectively reduce sycophancy in VLMs.

Findings

Higher model layers are better at preventing sycophancy.

Lack of image attention in higher layers may cause sycophancy.

Enhancing image attention at high layers mitigates sycophantic responses.

Abstract

In the study of LLMs, sycophancy represents a prevalent hallucination that poses significant challenges to these models. Specifically, LLMs often fail to adhere to original correct responses, instead blindly agreeing with users' opinions, even when those opinions are incorrect or malicious. However, research on sycophancy in visual language models (VLMs) has been scarce. In this work, we extend the exploration of sycophancy from LLMs to VLMs, introducing the MM-SY benchmark to evaluate this phenomenon. We present evaluation results from multiple representative models, addressing the gap in sycophancy research for VLMs. To mitigate sycophancy, we propose a synthetic dataset for training and employ methods based on prompts, supervised fine-tuning, and DPO. Our experiments demonstrate that these methods effectively alleviate sycophancy in VLMs. Additionally, we probe VLMs to assess the semantic impact of sycophancy and analyze the attention distribution of visual tokens. Our findings indicate that the ability to prevent sycophancy is predominantly observed in higher layers of the model. The lack of attention to image knowledge in these higher layers may contribute to sycophancy, and enhancing image attention at high layers proves beneficial in mitigating this issue.