Information Design in Multi-Agent Reinforcement Learning

TL;DR

This paper explores how information design influences multi-agent reinforcement learning, addressing challenges of non-stationarity and agent obedience, and introduces a new framework with practical algorithms for complex environments.

Contribution

It formulates the Markov signaling game for multi-agent RL, introduces signaling gradient and obedience constraints, and provides an efficient algorithm with insights into computational economics.

Findings

Algorithm performs well on mixed-motive tasks.

Provides a new framework for information influence in RL.

Offers insights into economic principles in multi-agent settings.

Abstract

Reinforcement learning (RL) is inspired by the way human infants and animals learn from the environment. The setting is somewhat idealized because, in actual tasks, other agents in the environment have their own goals and behave adaptively to the ego agent. To thrive in those environments, the agent needs to influence other agents so their actions become more helpful and less harmful. Research in computational economics distills two ways to influence others directly: by providing tangible goods (mechanism design) and by providing information (information design). This work investigates information design problems for a group of RL agents. The main challenges are two-fold. One is the information provided will immediately affect the transition of the agent trajectories, which introduces additional non-stationarity. The other is the information can be ignored, so the sender must provide information that the receiver is willing to respect. We formulate the Markov signaling game, and develop the notions of signaling gradient and the extended obedience constraints that address these challenges. Our algorithm is efficient on various mixed-motive tasks and provides further insights into computational economics. Our code is publicly available at https://github.com/YueLin301/InformationDesignMARL.