CREW-WILDFIRE: Benchmarking Agentic Multi-Agent Collaborations at Scale

TL;DR

CREW-Wildfire is a comprehensive benchmark designed to evaluate large-scale, multi-agent AI systems in complex wildfire response scenarios, addressing limitations of existing small-scale, low-complexity environments.

Contribution

It introduces a realistic, scalable wildfire response environment with diverse agents and tasks, enabling assessment of advanced multi-agent coordination and planning capabilities.

Findings

State-of-the-art LLM-based frameworks show significant performance gaps.

Highlights challenges in large-scale coordination and long-horizon planning.

Provides a foundation for future research in scalable multi-agent AI.

Abstract

Despite rapid progress in large language model (LLM)-based multi-agent systems, current benchmarks fall short in evaluating their scalability, robustness, and coordination capabilities in complex, dynamic, real-world tasks. Existing environments typically focus on small-scale, fully observable, or low-complexity domains, limiting their utility for developing and assessing next-generation multi-agent Agentic AI frameworks. We introduce CREW-Wildfire, an open-source benchmark designed to close this gap. Built atop the human-AI teaming CREW simulation platform, CREW-Wildfire offers procedurally generated wildfire response scenarios featuring large maps, heterogeneous agents, partial observability, stochastic dynamics, and long-horizon planning objectives. The environment supports both low-level control and high-level natural language interactions through modular Perception and Execution modules. We implement and evaluate several state-of-the-art LLM-based multi-agent Agentic AI frameworks, uncovering significant performance gaps that highlight the unsolved challenges in large-scale coordination, communication, spatial reasoning, and long-horizon planning under uncertainty. By providing more realistic complexity, scalable architecture, and behavioral evaluation metrics, CREW-Wildfire establishes a critical foundation for advancing research in scalable multi-agent Agentic intelligence. All code, environments, data, and baselines will be released to support future research in this emerging domain.