Generative Agents for Multi-Agent Autoformalization of Interaction Scenarios

TL;DR

This paper presents GAMA, a framework that automates the formalization of multi-agent interaction scenarios from natural language using large language models, enabling efficient simulation development with high correctness.

Contribution

GAMA introduces an automated method for formalizing interaction scenarios using LLMs, reducing manual effort and domain expertise needed for multi-agent simulations.

Findings

GAMA achieves 100% syntactic correctness in formalized scenarios.

Semantic correctness reaches up to 77% with GPT-4o.

High semantic accuracy in autoformalizing agent strategies.

Abstract

Multi-agent simulations are versatile tools for exploring interactions among natural and artificial agents, but their development typically demands domain expertise and manual effort. This work introduces the Generative Agents for Multi-Agent Autoformalization (GAMA) framework, which automates the formalization of interaction scenarios in simulations using agents augmented with large language models (LLMs). To demonstrate the application of GAMA, we use natural language descriptions of game-theoretic scenarios representing social interactions, and we autoformalize them into executable logic programs defining game rules, with syntactic correctness enforced through a solver-based validation. To ensure runtime validity, an iterative, tournament-based procedure tests the generated rules and strategies, followed by exact semantic validation when ground truth outcomes are available. In experiments with 110 natural language descriptions across five 2x2 simultaneous-move games, GAMA achieves 100% syntactic and 76.5% semantic correctness with Claude 3.5 Sonnet, and 99.82% syntactic and 77% semantic correctness with GPT-4o. The framework also shows high semantic accuracy in autoformalizing agents' strategies.