Advancing Language Multi-Agent Learning with Credit Re-Assignment for Interactive Environment Generalization

TL;DR

This paper introduces CollabUIAgents, a multi-agent reinforcement learning framework that uses a novel credit re-assignment strategy with LLMs to improve performance and generalization across diverse interactive environments.

Contribution

It proposes a new credit re-assignment method leveraging LLMs and synthesized preferences to enhance multi-agent system generalization and performance.

Findings

Improved cross-environment generalization of multi-agent systems.

A 7B-parameter system matching or surpassing strong closed-source models.

Insights into effective use of granular CR rewards for environment generalization.

Abstract

LLM-based agents have made significant advancements in interactive environments, such as mobile operations and web browsing, and other domains beyond computer using. Current multi-agent systems universally excel in performance, compared to single agents, but struggle with generalization across environments due to predefined roles and inadequate strategies for generalizing language agents. The challenge of achieving both strong performance and good generalization has hindered the progress of multi-agent systems for interactive environments. To address these issues, we propose CollabUIAgents, a multi-agent reinforcement learning framework with a novel multi-agent credit re-assignment (CR) strategy, assigning process rewards with LLMs rather than environment-specific rewards and learning with synthesized preference data, in order to foster generalizable, collaborative behaviors among the role-free agents' policies. Empirical results show that our framework improves both performance and cross-environment generalizability of multi-agent systems. Moreover, our 7B-parameter system achieves results on par with or exceed strong closed-source models, and the LLM that guides the CR. We also provide insights in using granular CR rewards effectively for environment generalization, and accommodating trained LLMs in multi-agent systems.