Structural Credit Assignment with Coordinated Exploration

TL;DR

This paper introduces a biologically plausible, coordinated exploration method for training neural networks that improves learning speed over traditional independent exploration, using Boltzmann machines or recurrent networks.

Contribution

It proposes using Boltzmann machines or recurrent networks for coordinated exploration, removing the negative phase, and demonstrating faster training compared to independent exploration methods.

Findings

Coordinated exploration significantly outperforms independent exploration in training speed.

The negative phase in Boltzmann machine training can be eliminated, simplifying the learning process.

The method surpasses STE backpropagation in training efficiency.

Abstract

A biologically plausible method for training an Artificial Neural Network (ANN) involves treating each unit as a stochastic Reinforcement Learning (RL) agent, thereby considering the network as a team of agents. Consequently, all units can learn via REINFORCE, a local learning rule modulated by a global reward signal, which aligns more closely with biologically observed forms of synaptic plasticity. However, this learning method tends to be slow and does not scale well with the size of the network. This inefficiency arises from two factors impeding effective structural credit assignment: (i) all units independently explore the network, and (ii) a single reward is used to evaluate the actions of all units. Accordingly, methods aimed at improving structural credit assignment can generally be classified into two categories. The first category includes algorithms that enable coordinated exploration among units, such as MAP propagation. The second category encompasses algorithms that compute a more specific reward signal for each unit within the network, like Weight Maximization and its variants. In this research report, our focus is on the first category. We propose the use of Boltzmann machines or a recurrent network for coordinated exploration. We show that the negative phase, which is typically necessary to train Boltzmann machines, can be removed. The resulting learning rules are similar to the reward-modulated Hebbian learning rule. Experimental results demonstrate that coordinated exploration significantly exceeds independent exploration in training speed for multiple stochastic and discrete units based on REINFORCE, even surpassing straight-through estimator (STE) backpropagation.