The Problem of Social Cost in Multi-Agent General Reinforcement Learning: Survey and Synthesis

TL;DR

This paper surveys and synthesizes approaches to measuring and controlling social harms caused by multi-agent reinforcement learning agents, proposing market-based mechanisms to address the open problem of social impact assessment in AI safety.

Contribution

It introduces a general framework for quantifying social costs in multi-agent reinforcement learning, extending existing models with history-based environments and diverse agent strategies.

Findings

Market-based mechanisms can effectively quantify social harms.

The framework encompasses well-studied special cases and extends existing models.

Applications include pollution control and the Paperclips problem.

Abstract

The AI safety literature is full of examples of powerful AI agents that, in blindly pursuing a specific and usually narrow objective, ends up with unacceptable and even catastrophic collateral damage to others. In this paper, we consider the problem of social harms that can result from actions taken by learning and utility-maximising agents in a multi-agent environment. The problem of measuring social harms or impacts in such multi-agent settings, especially when the agents are artificial generally intelligent (AGI) agents, was listed as an open problem in Everitt et al, 2018. We attempt a partial answer to that open problem in the form of market-based mechanisms to quantify and control the cost of such social harms. The proposed setup captures many well-studied special cases and is more general than existing formulations of multi-agent reinforcement learning with mechanism design in two ways: (i) the underlying environment is a history-based general reinforcement learning environment like in AIXI; (ii) the reinforcement-learning agents participating in the environment can have different learning strategies and planning horizons. To demonstrate the practicality of the proposed setup, we survey some key classes of learning algorithms and present a few applications, including a discussion of the Paperclips problem and pollution control with a cap-and-trade system.