Peer Learning: Learning Complex Policies in Groups from Scratch via Action Recommendations

TL;DR

This paper introduces peer learning, a reinforcement learning framework where groups of agents learn complex policies together through limited communication, outperforming traditional single-agent methods and baselines in various domains.

Contribution

It formalizes peer learning as a multi-armed bandit problem for teacher selection and demonstrates its effectiveness in learning complex policies in both discrete and continuous environments.

Findings

Peer learning outperforms single-agent and baseline methods.

Agents can rank peers' performance and identify reliable advice.

Complex policies can evolve from action recommendations beyond discrete actions.

Abstract

Peer learning is a novel high-level reinforcement learning framework for agents learning in groups. While standard reinforcement learning trains an individual agent in trial-and-error fashion, all on its own, peer learning addresses a related setting in which a group of agents, i.e., peers, learns to master a task simultaneously together from scratch. Peers are allowed to communicate only about their own states and actions recommended by others: "What would you do in my situation?". Our motivation is to study the learning behavior of these agents. We formalize the teacher selection process in the action advice setting as a multi-armed bandit problem and therefore highlight the need for exploration. Eventually, we analyze the learning behavior of the peers and observe their ability to rank the agents' performance within the study group and understand which agents give reliable advice. Further, we compare peer learning with single agent learning and a state-of-the-art action advice baseline. We show that peer learning is able to outperform single-agent learning and the baseline in several challenging discrete and continuous OpenAI Gym domains. Doing so, we also show that within such a framework complex policies from action recommendations beyond discrete action spaces can evolve.