Investigating the Scalability and Biological Plausibility of the Activation Relaxation Algorithm

TL;DR

This paper demonstrates that the Activation Relaxation algorithm, with biologically plausible modifications, scales effectively to complex CNN architectures and challenging datasets, maintaining performance while relaxing certain assumptions.

Contribution

It extends the Activation Relaxation algorithm's applicability to complex architectures and datasets, validating its biological plausibility and scalability.

Findings

Maintains performance on complex CNNs and datasets

Simplifications do not impair effectiveness

Relaxing the frozen feedforward pass is feasible

Abstract

The recently proposed Activation Relaxation (AR) algorithm provides a simple and robust approach for approximating the backpropagation of error algorithm using only local learning rules. Unlike competing schemes, it converges to the exact backpropagation gradients, and utilises only a single type of computational unit and a single backwards relaxation phase. We have previously shown that the algorithm can be further simplified and made more biologically plausible by (i) introducing a learnable set of backwards weights, which overcomes the weight-transport problem, and (ii) avoiding the computation of nonlinear derivatives at each neuron. However, tthe efficacy of these simplifications has, so far, only been tested on simple multi-layer-perceptron (MLP) networks. Here, we show that these simplifications still maintain performance using more complex CNN architectures and challenging datasets, which have proven difficult for other biologically-plausible schemes to scale to. We also investigate whether another biologically implausible assumption of the original AR algorithm -- the frozen feedforward pass -- can be relaxed without damaging performance.