Truthful Self-Play

TL;DR

This paper introduces a mechanism-inspired evolutionary learning framework that encourages truthful communication among agents in multi-agent reinforcement learning, leading to improved state representations and performance.

Contribution

It proposes a novel modification of self-play using peer prediction to promote truthful signaling, addressing local optima issues in non-cooperative environments.

Findings

Achieves state-of-the-art performance in multiple tasks

Effectively elicits truthful signals among agents

Improves emergent state representations

Abstract

We present a general framework for evolutionary learning to emergent unbiased state representation without any supervision. Evolutionary frameworks such as self-play converge to bad local optima in case of multi-agent reinforcement learning in non-cooperative partially observable environments with communication due to information asymmetry. Our proposed framework is a simple modification of self-play inspired by mechanism design, also known as {\em reverse game theory}, to elicit truthful signals and make the agents cooperative. The key idea is to add imaginary rewards using the peer prediction method, i.e., a mechanism for evaluating the validity of information exchanged between agents in a decentralized environment. Numerical experiments with predator prey, traffic junction and StarCraft tasks demonstrate that the state-of-the-art performance of our framework.