Distributed Partial Information Puzzles: Examining Common Ground Construction Under Epistemic Asymmetry

TL;DR

This paper introduces the Distributed Partial Information Puzzle (DPIP), a multimodal task designed to study how AI systems establish common ground under epistemic asymmetry, highlighting current limitations of large language models in belief tracking.

Contribution

The paper presents a novel multimodal dataset for common ground construction and compares LLMs with a DEL-based model, revealing challenges faced by LLMs in belief state tracking.

Findings

LLMs struggle to accurately track belief states in DPIP.

The DEL-based model effectively models belief dynamics.

DPIP reveals limitations of current AI in multimodal common ground reasoning.

Abstract

Establishing common ground, a shared set of beliefs and mutually recognized facts, is fundamental to collaboration, yet remains a challenge for current AI systems, especially in multimodal, multiparty settings, where the collaborators bring different information to the table. We introduce the Distributed Partial Information Puzzle (DPIP), a collaborative construction task that elicits rich multimodal communication under epistemic asymmetry. We present a multimodal dataset of these interactions, annotated and temporally aligned across speech, gesture, and action modalities to support reasoning over propositional content and belief dynamics. We then evaluate two paradigms for modeling common ground (CG): (1) state-of-the-art large language models (LLMs), prompted to infer shared beliefs from multimodal updates, and (2) an axiomatic pipeline grounded in Dynamic Epistemic Logic (DEL) that incrementally performs the same task. Results on the annotated DPIP data indicate that it poses a challenge to modern LLMs' abilities to track both task progression and belief state.