The Price of Agreement: Measuring LLM Sycophancy in Agentic Financial Applications

TL;DR

This paper evaluates the tendency of large language models to prioritize agreement over correctness in financial tasks, revealing their limited robustness and proposing benchmarks for measuring and improving their sycophantic behavior.

Contribution

It introduces new tasks to measure LLM sycophancy in financial settings and benchmarks recovery methods, highlighting the models' vulnerabilities.

Findings

Models show only modest performance drops with user rebuttals.

Most models fail when user preferences contradict reference answers.

Input filtering with pretrained LLMs can help recover from sycophantic responses.

Abstract

Given the increased use of LLMs in financial systems today, it becomes important to evaluate the safety and robustness of such systems. One failure mode that LLMs frequently display in general domain settings is that of sycophancy. That is, models prioritize agreement with expressed user beliefs over correctness, leading to decreased accuracy and trust. In this work, we focus on evaluating sycophancy that LLMs display in agentic financial tasks. Our findings are three-fold: first, we find the models show only low to modest drops in performance in the face of user rebuttals or contradictions to the reference answer, which distinguishes sycophancy that models display in financial agentic settings from findings in prior work. Second, we introduce a suite of tasks to test for sycophancy by user preference information that contradicts the reference answer and find that most models fail in the presence of such inputs. Lastly, we benchmark different modes of recovery such as input filtering with a pretrained LLM.