BeliefShift: Benchmarking Temporal Belief Consistency and Opinion Drift in LLM Agents

TL;DR

BeliefShift presents a new benchmark for evaluating how large language models manage belief consistency and opinion changes over multiple sessions, highlighting the trade-offs between personalization and factual grounding.

Contribution

It introduces a longitudinal benchmark with novel metrics to assess belief dynamics in multi-session LLM interactions across diverse domains.

Findings

Models balancing personalization and factual grounding exhibit trade-offs.

New metrics effectively quantify belief revision and contradiction resolution.

Evaluation across multiple models reveals strengths and weaknesses in belief management.

Abstract

LLMs are increasingly used as long-running conversational agents, yet every major benchmark evaluating their memory treats user information as static facts to be stored and retrieved. That's the wrong model. People change their minds, and over extended interactions, phenomena like opinion drift, over-alignment, and confirmation bias start to matter a lot. BeliefShift introduces a longitudinal benchmark designed specifically to evaluate belief dynamics in multi-session LLM interactions. It covers three tracks: Temporal Belief Consistency, Contradiction Detection, and Evidence-Driven Revision. The dataset includes 2,400 human-annotated multi-session interaction trajectories spanning health, politics, personal values, and product preferences. We evaluate seven models including GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, LLaMA-3, and Mistral-Large under zero-shot and retrieval-augmented generation (RAG) settings. Results reveal a clear trade-off: models that personalize aggressively resist drift poorly, while factually grounded models miss legitimate belief updates. We further introduce four novel evaluation metrics: Belief Revision Accuracy (BRA), Drift Coherence Score (DCS), Contradiction Resolution Rate (CRR), and Evidence Sensitivity Index (ESI).