Ask WhAI:Probing Belief Formation in Role-Primed LLM Agents

TL;DR

Ask WhAI is a framework for analyzing and testing belief states in multi-agent language models, revealing how priors and reasoning influence diagnostic decisions in complex scenarios.

Contribution

It introduces a systems-level approach to inspect, perturb, and understand belief formation in role-primed LLM agents during multi-agent interactions.

Findings

Agents mirror real-world disciplinary biases.

Beliefs often rely on canonical studies and resist counterevidence.

The framework enables detailed interrogation of belief dynamics.

Abstract

We present Ask WhAI, a systems-level framework for inspecting and perturbing belief states in multi-agent interactions. The framework records and replays agent interactions, supports out-of-band queries into each agent's beliefs and rationale, and enables counterfactual evidence injection to test how belief structures respond to new information. We apply the framework to a medical case simulator notable for its multi-agent shared memory (a time-stamped electronic medical record, or EMR) and an oracle agent (the LabAgent) that holds ground truth lab results revealed only when explicitly queried. We stress-test the system on a multi-specialty diagnostic journey for a child with an abrupt-onset neuropsychiatric presentation. Large language model agents, each primed with strong role-specific priors ("act like a neurologist", "act like an infectious disease specialist"), write to a shared medical record and interact with a moderator across sequential or parallel encounters. Breakpoints at key diagnostic moments enable pre- and post-event belief queries, allowing us to distinguish entrenched priors from reasoning or evidence-integration effects. The simulation reveals that agent beliefs often mirror real-world disciplinary stances, including overreliance on canonical studies and resistance to counterevidence, and that these beliefs can be traced and interrogated in ways not possible with human experts. By making such dynamics visible and testable, Ask WhAI offers a reproducible way to study belief formation and epistemic silos in multi-agent scientific reasoning.