Measuring Sycophancy of Language Models in Multi-turn Dialogues

TL;DR

This paper introduces SYCON Bench, a new benchmark for measuring sycophantic behavior in multi-turn dialogues of large language models, revealing how different tuning and prompting strategies influence conformity to user beliefs.

Contribution

The paper presents SYCON Bench, a novel multi-turn benchmark for evaluating sycophancy in LLMs, and provides comprehensive analysis of factors affecting model conformity in real-world scenarios.

Findings

Alignment tuning increases sycophantic behavior.

Scaling and reasoning improve resistance to user influence.

Third-person prompts reduce sycophancy by up to 63.8%.

Abstract

Large Language Models (LLMs) are expected to provide helpful and harmless responses, yet they often exhibit sycophancy--conforming to user beliefs regardless of factual accuracy or ethical soundness. Prior research on sycophancy has primarily focused on single-turn factual correctness, overlooking the dynamics of real-world interactions. In this work, we introduce SYCON Bench, a novel benchmark for evaluating sycophantic behavior in multi-turn, free-form conversational settings. Our benchmark measures how quickly a model conforms to the user (Turn of Flip) and how frequently it shifts its stance under sustained user pressure (Number of Flip). Applying SYCON Bench to 17 LLMs across three real-world scenarios, we find that sycophancy remains a prevalent failure mode. Our analysis shows that alignment tuning amplifies sycophantic behavior, whereas model scaling and reasoning optimization strengthen the model's ability to resist undesirable user views. Reasoning models generally outperform instruction-tuned models but often fail when they over-index on logical exposition instead of directly addressing the user's underlying beliefs. Finally, we evaluate four additional prompting strategies and demonstrate that adopting a third-person perspective reduces sycophancy by up to 63.8% in debate scenario. We release our code and data at https://github.com/JiseungHong/SYCON-Bench.