MAP Propagation Algorithm: Faster Learning with a Team of Reinforcement Learning Agents

TL;DR

The paper introduces MAP propagation, a novel algorithm that significantly reduces variance in reinforcement learning with neural networks, enabling faster and more biologically plausible training of deep networks by teams of agents.

Contribution

It proposes MAP propagation, a new variance reduction technique for local reinforcement learning rules in neural networks, improving training speed and biological plausibility.

Findings

MAP propagation matches backpropagation speed in RL tasks

It reduces variance of REINFORCE-based updates

Enables training of deep networks with local learning rules

Abstract

Nearly all state-of-the-art deep learning algorithms rely on error backpropagation, which is generally regarded as biologically implausible. An alternative way of training an artificial neural network is through treating each unit in the network as a reinforcement learning agent, and thus the network is considered as a team of agents. As such, all units can be trained by REINFORCE, a local learning rule modulated by a global signal that is more consistent with biologically observed forms of synaptic plasticity. Although this learning rule follows the gradient of return in expectation, it suffers from high variance and thus the low speed of learning, rendering it impractical to train deep networks. We therefore propose a novel algorithm called MAP propagation to reduce this variance significantly while retaining the local property of the learning rule. Experiments demonstrated that MAP propagation could solve common reinforcement learning tasks at a similar speed to backpropagation when applied to an actor-critic network. Our work thus allows for the broader application of the teams of agents in deep reinforcement learning.