DIFFER: Decomposing Individual Reward for Fair Experience Replay in Multi-Agent Reinforcement Learning

TL;DR

This paper introduces DIFFER, a theoretical framework that decomposes team rewards into individual rewards in multi-agent reinforcement learning, improving fairness and learning efficiency by leveraging gradient invariance and a PDE solution.

Contribution

The paper presents a novel reward decomposition method for MARL that enhances experience replay fairness and efficiency, supported by a new theoretical foundation and practical algorithms.

Findings

Significant improvements in learning efficiency on benchmarks.

Enhanced fairness in multi-agent experience replay.

Theoretical validation of reward decomposition approach.

Abstract

Cooperative multi-agent reinforcement learning (MARL) is a challenging task, as agents must learn complex and diverse individual strategies from a shared team reward. However, existing methods struggle to distinguish and exploit important individual experiences, as they lack an effective way to decompose the team reward into individual rewards. To address this challenge, we propose DIFFER, a powerful theoretical framework for decomposing individual rewards to enable fair experience replay in MARL. By enforcing the invariance of network gradients, we establish a partial differential equation whose solution yields the underlying individual reward function. The individual TD-error can then be computed from the solved closed-form individual rewards, indicating the importance of each piece of experience in the learning task and guiding the training process. Our method elegantly achieves an equivalence to the original learning framework when individual experiences are homogeneous, while also adapting to achieve more muscular efficiency and fairness when diversity is observed.Our extensive experiments on popular benchmarks validate the effectiveness of our theory and method, demonstrating significant improvements in learning efficiency and fairness.