MAS-on-the-Fly: Dynamic Adaptation of LLM-based Multi-Agent Systems at Test Time

TL;DR

MAS-on-the-Fly introduces a dynamic multi-agent system framework that adapts in real-time at test time, significantly improving task success rates and robustness in complex scenarios.

Contribution

It presents MASFly, a novel framework enabling real-time adaptation of LLM-based multi-agent systems through retrieval-augmented instantiation and experience-guided supervision.

Findings

Achieves 61.7% success rate on TravelPlanner benchmark.

Demonstrates strong task adaptability and robustness.

Outperforms existing static multi-agent systems.

Abstract

Large Language Model (LLM)-based multi-agent systems (MAS) have emerged as a promising paradigm for solving complex tasks. However, existing works often rely on manual designs or "one-size-fits-all" automation, lacking dynamic adaptability after deployment. Inspired by how biological systems adapt, we introduce MASFly, a novel multi-agent framework enabling dynamic adaptation at test time. To adapt system generation, MASFly employs a retrieval-augmented SOP instantiation mechanism that leverages a self-constructed repository of successful collaboration patterns, enabling the LLM to assemble customized MASs for new queries. For adaptive execution, MASFly incorporates an experience-guided supervision mechanism, where a dedicated Watcher agent monitors system behaviors with reference to a personalized experience pool and provides real-time interventions. Extensive experiments demonstrate that MASFly achieves state-of-the-art performance, most notably a 61.7% success rate on the TravelPlanner benchmark, while exhibiting strong task adaptability and robustness.