ResMAS: Resilience Optimization in LLM-based Multi-agent Systems

TL;DR

This paper introduces ResMAS, a framework that enhances the resilience of large language model-based multi-agent systems by optimizing communication topology and prompt design, leading to improved robustness against perturbations.

Contribution

The paper presents a novel two-stage framework that proactively improves MAS resilience through topology generation and prompt optimization, addressing a gap in existing reactive defense strategies.

Findings

ResMAS significantly improves MAS resilience across various tasks.

The topology generator effectively designs resilient communication structures.

Prompt optimization enhances agent cooperation and robustness.

Abstract

Large Language Model-based Multi-Agent Systems (LLM-based MAS), where multiple LLM agents collaborate to solve complex tasks, have shown impressive performance in many areas. However, MAS are typically distributed across different devices or environments, making them vulnerable to perturbations such as agent failures. While existing works have studied the adversarial attacks and corresponding defense strategies, they mainly focus on reactively detecting and mitigating attacks after they occur rather than proactively designing inherently resilient systems. In this work, we study the resilience of LLM-based MAS under perturbations and find that both the communication topology and prompt design significantly influence system resilience. Motivated by these findings, we propose ResMAS: a two-stage framework for enhancing MAS resilience. First, we train a reward model to predict the MAS's resilience, based on which we train a topology generator to automatically design resilient topology for specific tasks through reinforcement learning. Second, we introduce a topology-aware prompt optimization method that refines each agent's prompt based on its connections and interactions with other agents. Extensive experiments across a range of tasks show that our approach substantially improves MAS resilience under various constraints. Moreover, our framework demonstrates strong generalization ability to new tasks and models, highlighting its potential for building resilient MASs.