Local Pairwise Distance Matching for Backpropagation-Free Reinforcement Learning

TL;DR

This paper introduces a backpropagation-free reinforcement learning method that trains neural network layers using local pairwise distance matching, improving stability and performance without requiring backward passes.

Contribution

It presents a novel layer-wise training approach using local signals based on pairwise distance matching, eliminating the need for backpropagation in RL.

Findings

Achieves competitive performance with traditional backpropagation methods.

Enhances stability and consistency across training runs.

Improves learning in challenging RL environments.

Abstract

Training neural networks with reinforcement learning (RL) typically relies on backpropagation (BP), necessitating storage of activations from the forward pass for subsequent backward updates. Furthermore, backpropagating error signals through multiple layers often leads to vanishing or exploding gradients, which can degrade learning performance and stability. We propose a novel approach that trains each layer of the neural network using local signals during the forward pass in RL settings. Our approach introduces local, layer-wise losses leveraging the principle of matching pairwise distances from multi-dimensional scaling, enhanced with optional reward-driven guidance. This method allows each hidden layer to be trained using local signals computed during forward propagation, thus eliminating the need for backward passes and storing intermediate activations. Our experiments, conducted with policy gradient methods across common RL benchmarks, demonstrate that this backpropagation-free method achieves competitive performance compared to their classical BP-based counterpart. Additionally, the proposed method enhances stability and consistency within and across runs, and improves performance especially in challenging environments.