Who Do LLMs Trust? Human Experts Matter More Than Other LLMs

TL;DR

This study shows that large language models are more influenced by human expert feedback than by other LLMs, indicating a credibility-based social influence similar to humans across various decision tasks.

Contribution

It demonstrates that instruction-tuned LLMs prioritize human expert input over other LLMs, revealing a credibility-sensitive influence pattern in AI models.

Findings

Models conform more to human expert responses, even when incorrect.

Models revise answers more towards experts than other LLMs.

Expert framing strongly influences LLM decision-making.

Abstract

Large language models (LLMs) increasingly operate in environments where they encounter social information such as other agents' answers, tool outputs, or human recommendations. In humans, such inputs influence judgments in ways that depend on the source's credibility and the strength of consensus. This paper investigates whether LLMs exhibit analogous patterns of influence and whether they privilege feedback from humans over feedback from other LLMs. Across three binary decision-making tasks, reading comprehension, multi-step reasoning, and moral judgment, we present four instruction-tuned LLMs with prior responses attributed either to friends, to human experts, or to other LLMs. We manipulate whether the group is correct and vary the group size. In a second experiment, we introduce direct disagreement between a single human and a single LLM. Across tasks, models conform significantly more to responses labeled as coming from human experts, including when that signal is incorrect, and revise their answers toward experts more readily than toward other LLMs. These results reveal that expert framing acts as a strong prior for contemporary LLMs, suggesting a form of credibility-sensitive social influence that generalizes across decision domains.