$\mu$PC: Scaling Predictive Coding to 100+ Layer Networks

TL;DR

This paper demonstrates that predictive coding networks can be scaled to over 100 layers using a new parameterization, achieving stable training and competitive performance on simple tasks, thus advancing biologically plausible learning algorithms.

Contribution

The authors introduce $$PC, a scalable predictive coding method using Depth-$$p parameterization, enabling training of very deep PC networks with stability and transferability.

Findings

Stable training of 128-layer residual PC networks achieved

Zero-shot transfer of learning rates across widths and depths demonstrated

Predictive coding networks can reach competitive performance on simple classification tasks

Abstract

The biological implausibility of backpropagation (BP) has motivated many alternative, brain-inspired algorithms that attempt to rely only on local information, such as predictive coding (PC) and equilibrium propagation. However, these algorithms have notoriously struggled to train very deep networks, preventing them from competing with BP in large-scale settings. Indeed, scaling PC networks (PCNs) has recently been posed as a challenge for the community (Pinchetti et al., 2024). Here, we show that 100+ layer PCNs can be trained reliably using a Depth-$\mu$P parameterisation (Yang et al., 2023; Bordelon et al., 2023) which we call "$\mu$PC". By analysing the scaling behaviour of PCNs, we reveal several pathologies that make standard PCNs difficult to train at large depths. We then show that, despite addressing only some of these instabilities, $\mu$PC allows stable training of very deep (up to 128-layer) residual networks on simple classification tasks with competitive performance and little tuning compared to current benchmarks. Moreover, $\mu$PC enables zero-shot transfer of both weight and activity learning rates across widths and depths. Our results serve as a first step towards scaling PC to more complex architectures and have implications for other local algorithms. Code for $\mu$PC is made available as part of a JAX library for PCNs.