From Particles to Agents: Hallucination as a Metric for Cognitive Friction in Spatial Simulation

TL;DR

This paper introduces a novel framework for spatial simulation using multimodal generative models that predict environment states through episodic reasoning, leveraging AI hallucinations to diagnose semiotic ambiguities and enhance human-environment interaction.

Contribution

It proposes Agentic Environmental Simulations with episodic spatial reasoning, formalizes Cognitive Friction as a diagnostic tool, and redefines environment interaction as dynamic cognitive partnership.

Findings

Hallucinations serve as diagnostic tools for semiotic ambiguities.

Episodic spatial reasoning improves simulation meaningfulness.

Framework supports human-centered AI-driven environment design.

Abstract

Traditional architectural simulations (e.g. Computational Fluid Dynamics, evacuation, structural analysis) model elements as deterministic physics-based "particles" rather than cognitive "agents". To bridge this, we introduce \textbf{Agentic Environmental Simulations}, where Large Multimodal generative models actively predict the next state of spatial environments based on semantic expectation. Drawing on examples from accessibility-oriented AR pipelines and multimodal digital twins, we propose a shift from chronological time-steps to Episodic Spatial Reasoning, where simulations advance through meaningful, surprisal-triggered events. Within this framework we posit AI hallucinations as diagnostic tools. By formalizing the \textbf{Cognitive Friction} ($C_f$) it is possible to reveal "Phantom Affordances", i.e. semiotic ambiguities in built space. Finally, we challenge current HCI paradigms by treating environments as dynamic cognitive partners and propose a human-centered framework of cognitive orchestration for designing AI-driven simulations that preserve autonomy, affective clarity, and cognitive integrity.