Learning to Teach in Cooperative Multiagent Reinforcement Learning

TL;DR

This paper introduces LeCTR, a novel algorithm enabling cooperative multiagent reinforcement learning agents to learn when and what to teach each other, significantly improving learning speed and coordination without prior domain knowledge.

Contribution

It presents the first general framework and algorithm for agents to learn to teach and coordinate in multiagent reinforcement learning environments.

Findings

LeCTR agents learn significantly faster than state-of-the-art methods.

Agents effectively learn when to teach and when to be students.

LeCTR improves coordination in complex multiagent tasks.

Abstract

Collective human knowledge has clearly benefited from the fact that innovations by individuals are taught to others through communication. Similar to human social groups, agents in distributed learning systems would likely benefit from communication to share knowledge and teach skills. The problem of teaching to improve agent learning has been investigated by prior works, but these approaches make assumptions that prevent application of teaching to general multiagent problems, or require domain expertise for problems they can apply to. This learning to teach problem has inherent complexities related to measuring long-term impacts of teaching that compound the standard multiagent coordination challenges. In contrast to existing works, this paper presents the first general framework and algorithm for intelligent agents to learn to teach in a multiagent environment. Our algorithm, Learning to Coordinate and Teach Reinforcement (LeCTR), addresses peer-to-peer teaching in cooperative multiagent reinforcement learning. Each agent in our approach learns both when and what to advise, then uses the received advice to improve local learning. Importantly, these roles are not fixed; these agents learn to assume the role of student and/or teacher at the appropriate moments, requesting and providing advice in order to improve teamwide performance and learning. Empirical comparisons against state-of-the-art teaching methods show that our teaching agents not only learn significantly faster, but also learn to coordinate in tasks where existing methods fail.