A Policy Resonance Approach to Solve the Problem of Responsibility Diffusion in Multiagent Reinforcement Learning

TL;DR

This paper introduces a Policy Resonance approach to address Responsibility Diffusion in multiagent reinforcement learning, enhancing collaboration by refactoring joint policies without altering individual behaviors, proven effective in benchmark tasks.

Contribution

The paper presents a novel Policy Resonance method that refactors joint policies to improve collaboration and responsibility sharing in multiagent systems, addressing a key limitation of existing algorithms.

Findings

Policy Resonance improves cooperative performance in benchmarks.

Addresses Responsibility Diffusion by refactoring joint policies.

Compatible with state-of-the-art multiagent algorithms.

Abstract

SOTA multiagent reinforcement algorithms distinguish themselves in many ways from their single-agent equivalences. However, most of them still totally inherit the single-agent exploration-exploitation strategy. Naively inheriting this strategy from single-agent algorithms causes potential collaboration failures, in which the agents blindly follow mainstream behaviors and reject taking minority responsibility. We name this problem the Responsibility Diffusion (RD) as it shares similarities with a same-name social psychology effect. In this work, we start by theoretically analyzing the cause of this RD problem, which can be traced back to the exploration-exploitation dilemma of multiagent systems (especially large-scale multiagent systems). We address this RD problem by proposing a Policy Resonance (PR) approach which modifies the collaborative exploration strategy of agents by refactoring the joint agent policy while keeping individual policies approximately invariant. Next, we show that SOTA algorithms can equip this approach to promote the collaborative performance of agents in complex cooperative tasks. Experiments are performed in multiple test benchmark tasks to illustrate the effectiveness of this approach.