AIDG: Evaluating Asymmetry Between Information Extraction and Containment in Multi-Turn Dialogue

TL;DR

This paper introduces AIDG, a game-theoretic framework to evaluate LLMs' abilities in dynamic dialogue, revealing a significant asymmetry favoring containment over active deduction in multi-turn interactions.

Contribution

The paper presents AIDG, a novel framework with two tasks to measure information deduction and containment, highlighting the asymmetry in LLMs' strategic reasoning capabilities.

Findings

Models perform better at containment than deduction.

Confirmation strategies are significantly more effective than blind deduction.

Instruction-following degrades under conversational load, affecting deduction.

Abstract

Evaluating the strategic reasoning capabilities of Large Language Models (LLMs) requires moving beyond static benchmarks to dynamic, multi-turn interactions. We introduce AIDG (Adversarial Information Deduction Game), a game-theoretic framework that probes the asymmetry between information extraction (active deduction) and information containment (state maintenance) in dialogue. We propose two complementary tasks: AIDG-I, measuring pragmatic strategy in social deduction, and AIDG-II, measuring constraint satisfaction in a structured "20 Questions" setting. Across 439 games with six frontier LLMs, we observe a clear capability asymmetry: models perform substantially better at containment than deduction, with a 350 ELO advantage on defense;(Cohen's d = 5.47). We identify two bottlenecks driving this gap: (1) Information Dynamics, where confirmation strategies are 7.75x more effective than blind deduction (p < 0.00001), and (2) Constraint Adherence, where instruction-following degrades under conversational load, accounting for 41.3% of deductive failures. These findings suggest that while LLMs excel at local defensive coherence, they struggle with the global state tracking required for strategic inquiry.