Explicit Credit Assignment through Local Rewards and Dependence Graphs in Multi-Agent Reinforcement Learning

TL;DR

This paper introduces a method that combines local rewards and dependence graphs to improve credit assignment and cooperation in multi-agent reinforcement learning, balancing the benefits of both approaches.

Contribution

It proposes a novel approach using interaction graphs to better discern individual contributions while maintaining the advantages of local rewards.

Findings

Improves cooperation over traditional reward methods

Balances local and global reward benefits

Demonstrates flexibility and effectiveness in experiments

Abstract

To promote cooperation in Multi-Agent Reinforcement Learning, the reward signals of all agents can be aggregated together, forming global rewards that are commonly known as the fully cooperative setting. However, global rewards are usually noisy because they contain the contributions of all agents, which have to be resolved in the credit assignment process. On the other hand, using local reward benefits from faster learning due to the separation of agents' contributions, but can be suboptimal as agents myopically optimize their own reward while disregarding the global optimality. In this work, we propose a method that combines the merits of both approaches. By using a graph of interaction between agents, our method discerns the individual agent contribution in a more fine-grained manner than a global reward, while alleviating the cooperation problem with agents' local reward. We also introduce a practical approach for approximating such a graph. Our experiments demonstrate the flexibility of the approach, enabling improvements over the traditional local and global reward settings.