Bottom-Up Reputation Promotes Cooperation with Multi-Agent Reinforcement Learning

TL;DR

This paper introduces LR2, a bottom-up reputation learning method that enhances cooperation in multi-agent systems by shaping rewards based on local reputation assessments without relying on predefined norms.

Contribution

The paper proposes a novel bottom-up reputation learning approach, LR2, which promotes cooperation through local reputation-based reward shaping without centralized control or predefined norms.

Findings

LR2 effectively promotes cooperation across various social dilemma scenarios.

LR2 stabilizes cooperation and fosters strategy clustering in structured populations.

Reputation-based reward shaping improves cooperation without centralized norms.

Abstract

Reputation serves as a powerful mechanism for promoting cooperation in multi-agent systems, as agents are more inclined to cooperate with those of good social standing. While existing multi-agent reinforcement learning methods typically rely on predefined social norms to assign reputations, the question of how a population reaches a consensus on judgement when agents hold private, independent views remains unresolved. In this paper, we propose a novel bottom-up reputation learning method, Learning with Reputation Reward (LR2), designed to promote cooperative behaviour through rewards shaping based on assigned reputation. Our agent architecture includes a dilemma policy that determines cooperation by considering the impact on neighbours, and an evaluation policy that assigns reputations to affect the actions of neighbours while optimizing self-objectives. It operates using local observations and interaction-based rewards, without relying on centralized modules or predefined norms. Our findings demonstrate the effectiveness and adaptability of LR2 across various spatial social dilemma scenarios. Interestingly, we find that LR2 stabilizes and enhances cooperation not only with reward reshaping from bottom-up reputation but also by fostering strategy clustering in structured populations, thereby creating environments conducive to sustained cooperation.