Automated Adversarial Collaboration for Advancing Theory Building in the Cognitive Sciences

TL;DR

This paper presents an automated framework using AI and information theory to evaluate and compare cognitive science theories through a closed-loop simulation process.

Contribution

It introduces a novel automated adversarial collaboration system combining LLMs, program synthesis, and experimental design for theory adjudication.

Findings

Successfully identified the ground-truth theory in simulations

Recovered the correct theory across various noise levels

Demonstrated proof of concept for in-silico theory evaluation

Abstract

Cognitive science often evaluates theories through narrow paradigms and local model comparisons, limiting the integration of evidence across tasks and realizations. We introduce an automated adversarial collaboration framework for adjudicating among competing theories even when the candidate models and experiments must be discovered during the adjudication process. The system combines LLM-based theory agents, program synthesis, and information-theoretic experimental design in a closed loop. In a simulation study spanning three classic categorization theories, the framework recovered the ground-truth theory across noise settings with weaker reliability in the hardest settings. Together, the framework and findings provide a concrete proof of concept for closed-loop, in-silico theory adjudication in cognitive science.