What Evidence Do Language Models Find Convincing?

TL;DR

This paper investigates how retrieval-augmented language models evaluate conflicting evidence, revealing their reliance on relevance over stylistic features, and emphasizes the importance of high-quality RAG data for better alignment with human judgments.

Contribution

The study introduces the ConflictingQA dataset and analyzes LLM sensitivities to different evidence features, highlighting current model biases and suggesting improvements for RAG training.

Findings

Models rely heavily on relevance of evidence

Stylistic features like scientific references are largely ignored

High-quality, filtered RAG data is crucial for better alignment

Abstract

Retrieval-augmented language models are being increasingly tasked with subjective, contentious, and conflicting queries such as "is aspartame linked to cancer". To resolve these ambiguous queries, one must search through a large range of websites and consider "which, if any, of this evidence do I find convincing?". In this work, we study how LLMs answer this question. In particular, we construct ConflictingQA, a dataset that pairs controversial queries with a series of real-world evidence documents that contain different facts (e.g., quantitative results), argument styles (e.g., appeals to authority), and answers (Yes or No). We use this dataset to perform sensitivity and counterfactual analyses to explore which text features most affect LLM predictions. Overall, we find that current models rely heavily on the relevance of a website to the query, while largely ignoring stylistic features that humans find important such as whether a text contains scientific references or is written with a neutral tone. Taken together, these results highlight the importance of RAG corpus quality (e.g., the need to filter misinformation), and possibly even a shift in how LLMs are trained to better align with human judgements.