On the Infinite Width and Depth Limits of Predictive Coding Networks

TL;DR

This paper investigates the theoretical limits of predictive coding networks, showing that under certain conditions they behave like backpropagation in very wide and deep models, with implications for scalable neural network training.

Contribution

It provides a theoretical analysis of the infinite width and depth limits of predictive coding networks, establishing their equivalence to backpropagation under specific conditions.

Findings

Predictive coding networks converge to backpropagation in wide, deep regimes.

Theoretical unification of previous empirical results on PCNs.

Experimental evidence supports theoretical predictions in nonlinear networks.

Abstract

Predictive coding (PC) is a biologically plausible alternative to standard backpropagation (BP) that minimises an energy function with respect to network activities before updating weights. Recent work has improved the training stability of deep PC networks (PCNs) by leveraging some BP-inspired reparameterisations. However, the full scalability and theoretical basis of these approaches remains unclear. To address this, we study the infinite width and depth limits of PCNs. For linear residual networks, we show that the set of width- and depth-stable feature-learning parameterisations for PC is exactly the same as for BP. Moreover, under any of these parameterisations, the PC energy with equilibrated activities converges to the BP loss in a regime where the model width is much larger than the depth, resulting in PC computing the same gradients as BP. Experiments show that these results hold in practice for deep nonlinear networks, as long as an activity equilibrium seem to be reached. Overall, this work unifies various previous theoretical and empirical results and has potentially important implications for the scaling of PCNs.