Verification Required: The Impact of Information Credibility on AI Persuasion

TL;DR

This paper introduces MixTalk, a strategic communication game modeling information credibility in LLM interactions, and proposes TOPD to enhance robustness against persuasion in high-stakes communication scenarios.

Contribution

It presents a novel game framework for LLM communication with probabilistic credibility and introduces TOPD, an offline distillation method to improve robustness against persuasion.

Findings

TOPD significantly enhances receiver robustness to persuasion.

State-of-the-art LLM agents show strengths and limitations in reasoning about credibility.

MixTalk effectively models the strategic use of verifiable and unverifiable claims.

Abstract

Agents powered by large language models (LLMs) are increasingly deployed in settings where communication shapes high-stakes decisions, making a principled understanding of strategic communication essential. Prior work largely studies either unverifiable cheap-talk or fully verifiable disclosure, failing to capture realistic domains in which information has probabilistic credibility. We introduce MixTalk, a strategic communication game for LLM-to-LLM interaction that models information credibility. In MixTalk, a sender agent strategically combines verifiable and unverifiable claims to communicate private information, while a receiver agent allocates a limited budget to costly verification and infers the underlying state from prior beliefs, claims, and verification outcomes. We evaluate state-of-the-art LLM agents in large-scale tournaments across three realistic deployment settings, revealing their strengths and limitations in reasoning about information credibility and the explicit behavior that shapes these interactions. Finally, we propose Tournament Oracle Policy Distillation (TOPD), an offline method that distills tournament oracle policy from interaction logs and deploys it in-context at inference time. Our results show that TOPD significantly improves receiver robustness to persuasion.