How AI Responses Shape User Beliefs: The Effects of Information Detail and Confidence on Belief Strength and Stance

TL;DR

This study investigates how AI response features like detail and confidence influence user belief changes, revealing that medium confidence and detailed responses significantly impact belief strength and stance, with implications for AI system design and ethics.

Contribution

The paper introduces a novel analysis framework with belief switch and belief shift measures to quantify subtle belief adjustments caused by AI responses.

Findings

Detailed responses with medium confidence cause the largest belief changes.

High confidence responses lead to belief shifts but fewer reversals.

Task type and prior conviction modulate belief change dynamics.

Abstract

The growing use of AI-generated responses in everyday tools raises concern about how subtle features such as supporting detail or tone of confidence may shape people's beliefs. To understand this, we conducted a pre-registered online experiment (N = 304) investigating how the detail and confidence of AI-generated responses influence belief change. We introduce an analysis framework with two targeted measures: belief switch and belief shift. These distinguish between users changing their initial stance after AI input and the extent to which they adjust their conviction toward or away from the AI's stance, thereby quantifying not only categorical changes but also more subtle, continuous adjustments in belief strength that indicate a reinforcement or weakening of existing beliefs. Using this framework, we find that detailed responses with medium confidence are associated with the largest overall belief changes. Highly confident messages tend to elicit belief shifts but induce fewer stance reversals. Our results also show that task type (fact-checking versus opinion evaluation), prior conviction, and perceived stance agreement further modulate the extent and direction of belief change. These findings illustrate how different properties of AI responses interact with user beliefs in subtle but potentially consequential ways and raise practical as well as ethical considerations for the design of LLM-powered systems.