Kindness in Multi-Agent Reinforcement Learning

TL;DR

This paper introduces KindMARL, a novel multi-agent reinforcement learning method that incorporates fairness and kindness by evaluating agents' intentions through counterfactual reasoning, leading to improved cooperation and higher rewards.

Contribution

The paper proposes a new approach to MARL that models kindness via intention measurement using counterfactual analysis, enhancing cooperative behavior among agents.

Findings

Agents trained with KindMARL outperform existing methods in reward accumulation.

KindMARL significantly improves cooperation in environmental tasks.

Experimental results show increased total rewards in multiple environments.

Abstract

In human societies, people often incorporate fairness in their decisions and treat reciprocally by being kind to those who act kindly. They evaluate the kindness of others' actions not only by monitoring the outcomes but also by considering the intentions. This behavioral concept can be adapted to train cooperative agents in Multi-Agent Reinforcement Learning (MARL). We propose the KindMARL method, where agents' intentions are measured by counterfactual reasoning over the environmental impact of the actions that were available to the agents. More specifically, the current environment state is compared with the estimation of the current environment state provided that the agent had chosen another action. The difference between each agent's reward, as the outcome of its action, with that of its fellow, multiplied by the intention of the fellow is then taken as the fellow's "kindness". If the result of each reward-comparison confirms the agent's superiority, it perceives the fellow's kindness and reduces its own reward. Experimental results in the Cleanup and Harvest environments show that training based on the KindMARL method enabled the agents to earn 89\% (resp. 37\%) and 44% (resp. 43\%) more total rewards than training based on the Inequity Aversion and Social Influence methods. The effectiveness of KindMARL is further supported by experiments in a traffic light control problem.