PolicySimEval: A Benchmark for Evaluating Policy Outcomes through Agent-Based Simulation

TL;DR

PolicySimEval is a comprehensive benchmark designed to evaluate the effectiveness of agent-based simulation models in complex policy assessment tasks, highlighting current limitations and guiding future improvements.

Contribution

This paper introduces PolicySimEval, the first benchmark for systematically assessing agent-based models in policy evaluation, covering diverse scenarios and sub-tasks.

Findings

State-of-the-art systems achieve low coverage on benchmark tasks.

Current models struggle with complex policy simulation scenarios.

Benchmark reveals significant gaps in existing simulation capabilities.

Abstract

With the growing adoption of agent-based models in policy evaluation, a pressing question arises: Can such systems effectively simulate and analyze complex social scenarios to inform policy decisions? Addressing this challenge could significantly enhance the policy-making process, offering researchers and practitioners a systematic way to validate, explore, and refine policy outcomes. To advance this goal, we introduce PolicySimEval, the first benchmark designed to evaluate the capability of agent-based simulations in policy assessment tasks. PolicySimEval aims to reflect the real-world complexities faced by social scientists and policymakers. The benchmark is composed of three categories of evaluation tasks: (1) 20 comprehensive scenarios that replicate end-to-end policy modeling challenges, complete with annotated expert solutions; (2) 65 targeted sub-tasks that address specific aspects of agent-based simulation (e.g., agent behavior calibration); and (3) 200 auto-generated tasks to enable large-scale evaluation and method development. Experiments show that current state-of-the-art frameworks struggle to tackle these tasks effectively, with the highest-performing system achieving only 24.5\% coverage rate on comprehensive scenarios, 15.04\% on sub-tasks, and 14.5\% on auto-generated tasks. These results highlight the difficulty of the task and the gap between current capabilities and the requirements for real-world policy evaluation.