LOPT: Learning Optimal Pigovian Tax in Sequential Social Dilemmas

TL;DR

This paper introduces LOPT, a reinforcement learning method that learns optimal Pigovian taxes to internalize externalities, improving social welfare in multi-agent systems facing social dilemmas.

Contribution

The paper proposes LOPT, a novel approach that learns optimal externality-based taxes to mitigate social dilemmas in multi-agent reinforcement learning.

Findings

LOPT outperforms existing methods in Escape Room environment.

LOPT achieves higher collective social welfare in Cleanup environment.

The method effectively reduces social costs and alleviates social dilemmas.

Abstract

In multi-agent reinforcement learning, each agent acts to maximize its individual accumulated rewards. Nevertheless, individual accumulated rewards could not fully reflect how others perceive them, resulting in selfish behaviors that undermine global performance. The externality theory, defined as ``the activities of one economic actor affect the activities of another in ways that are not reflected in market transactions,'' is applicable to analyze the social dilemmas in MARL. One of its most profound non-market solutions, ``Pigovian Tax'', which internalizes externalities by taxing those who create negative externalities and subsidizing those who create positive externalities, could aid in developing a mechanism to resolve MARL's social dilemmas. The purpose of this paper is to apply externality theory to analyze social dilemmas in MARL. To internalize the externalities in MARL, the \textbf{L}earning \textbf{O}ptimal \textbf{P}igovian \textbf{T}ax method (LOPT), is proposed, where an additional agent is introduced to learn the tax/allowance allocation policy so as to approximate the optimal ``Pigovian Tax'' which accurately reflects the externalities for all agents. Furthermore, a reward shaping mechanism based on the approximated optimal ``Pigovian Tax'' is applied to reduce the social cost of each agent and tries to alleviate the social dilemmas. Compared with existing state-of-the-art methods, the proposed LOPT leads to higher collective social welfare in both the Escape Room and the Cleanup environments, which shows the superiority of our method in solving social dilemmas.