Are you with me? A Framework for Detecting Mental Model Discrepancies in Task-Based Team Dialogues

TL;DR

This paper introduces a framework to detect and categorize mental model discrepancies in team dialogues, aiming to improve real-time assessment of team coordination and predict future misalignments.

Contribution

It presents a novel method for real-time detection and categorization of four types of mental model discrepancies directly from team dialogues.

Findings

Discrepancy patterns contain predictive signals for future misalignments.

Averaging historical discrepancy counts yields meaningful prediction accuracy.

Different discrepancy types vary in their predictability.

Abstract

Humans typically use natural language to update teammates on task states. Since not all updates are communicated, discrepancies arise between the team members' mental models that negatively affect overall team performance. How can we categorize such discrepancies? Do misalignments detected in team dialogue predict future mental model misalignments? Traditional shared mental model (SMM) assessment methods rely on retrospective expert coding that cannot capture real-time coordination dynamics. We propose a framework to identify and categorize four types of mental model discrepancies: unsupported beliefs, false beliefs, belief contradictions, and omissions, all of which can naturally emerge in team dialogues. Using dialogues from twenty dyad teams performing collaborative object identification tasks across four sequential levels, we demonstrate that these discrepancy patterns contain predictive signals. Averaging historical discrepancy counts achieves meaningful prediction accuracy using uniform weighting as an exploratory baseline, with differential predictability across discrepancy types.