Shaping the learning signal in a combined Q-learning rule to improve structured cooperation

TL;DR

This paper investigates how shaping the learning signal with reputation influences cooperation in reinforcement learning, revealing that the effect depends on learning rate and discount factor, and can promote cooperation without altering game structure.

Contribution

It introduces a novel approach of combining reputation with payoffs in the learning signal, isolating its role in promoting cooperation within standard reinforcement learning frameworks.

Findings

Increasing reputation weight generally promotes cooperation.

The promoting effect vanishes with very small learning rates or high discount factors.

Reputation's influence is modulated by learning dynamics and parameters.

Abstract

Q-learning provides a standard reinforcement learning framework for studying cooperation by specifying how agents update action values from repeated local interactions outcomes. Although previous work has shown that reputation can promote cooperation in such systems, most models introduce reputation by modifying payoffs, encoding it directly in the state or changing partner selection, which makes it difficult to isolate the role of the learning signal itself. Here, we construct the reinforcement signal as a weighted combination of reputation and game payoffs, leaving the game and network structure unchanged. We find that increasing the weight on reputation generally promotes cooperation by consolidating clusters, but this effect is conditional on the learning dynamics. Specifically, this promoting effect vanishes in two regimes: when the learning rate is extremely small, which prevents effective information propagation and when the discount factor approaches one, as distant future expectations obscure the immediate reputational advantage. Outside these limiting cases, the efficacy of reputation in promoting cooperation is attenuated by higher learning rates but amplified by larger discount factors. These results advance the understanding of cooperative dynamics by demonstrating that cooperation can be stabilized through the reputational shaping of learning signals alone, providing critical insights into the interplay between social information and individual learning parameters.