Learning to Communicate to Solve Riddles with Deep Distributed Recurrent Q-Networks

TL;DR

This paper introduces DDRQN, a deep reinforcement learning approach enabling multi-agent teams to autonomously develop communication protocols to solve riddles, marking the first success in learning communication protocols with deep RL.

Contribution

The paper presents DDRQN, a novel deep distributed recurrent Q-network architecture that enables agents to learn communication protocols from scratch in multi-agent tasks.

Findings

DDRQN successfully solves communication-based riddles.

Agents develop effective, elegant communication protocols.

Each component of DDRQN is critical for its success.

Abstract

We propose deep distributed recurrent Q-networks (DDRQN), which enable teams of agents to learn to solve communication-based coordination tasks. In these tasks, the agents are not given any pre-designed communication protocol. Therefore, in order to successfully communicate, they must first automatically develop and agree upon their own communication protocol. We present empirical results on two multi-agent learning problems based on well-known riddles, demonstrating that DDRQN can successfully solve such tasks and discover elegant communication protocols to do so. To our knowledge, this is the first time deep reinforcement learning has succeeded in learning communication protocols. In addition, we present ablation experiments that confirm that each of the main components of the DDRQN architecture are critical to its success.