Mechanism Plausibility in Generative Agent-Based Modeling

TL;DR

This paper introduces a four-level Mechanism Plausibility Scale to evaluate how well generative models, especially LLM-based agent models, can produce and explain phenomena by distinguishing capability from mechanistic plausibility.

Contribution

It operationalizes a novel plausibility scale grounded in philosophy of science to differentiate between generative sufficiency and mechanistic explanation in agent-based modeling.

Findings

Developed a four-level scale for plausibility assessment

Clarified the distinction between capability and explanation in models

Integrated philosophy of science with LLM-ABM evaluation

Abstract

Large language models (LLMs) can generate high-level diverse phenomena without explicitly programmed rules. This capability has led to their adoption within different agent-based models (ABMs) and social simulations. Recent studies investigate their ability to generate different phenomena of interest, for example, human behavior on social media platforms or alien behavior in game-theoretic scenarios. However, capability, prediction, and explanation are different--drawing from the philosophy of science and mechanisms literature, explanation requires showing, to some degree, how a phenomenon is produced by related organized entities and activities. For modelers, describing the characteristics of an experiment or whether a simulation provides progress in capability (or explanation), can be difficult without being grounded in potentially distant research areas. We integrate recent work on LLM-ABMs with contemporary philosophy of science literature and use it to operationalize a definition of 'plausibility' in a four-level scale. Our scale separates the evaluation of a model's generative sufficiency (ability to reproduce a phenomenon) from its mechanistic plausibility (how the phenomenon could be produced), and clarifies the distinct roles of different models, such as predictive and explanatory ones. We introduce this as the Mechanism Plausibility Scale.