Tight Stability, Convergence, and Robustness Bounds for Predictive Coding Networks

TL;DR

This paper provides a rigorous dynamical systems analysis of predictive coding networks, demonstrating their stability, robustness, and convergence properties, and comparing their behavior to backpropagation and target propagation methods.

Contribution

It offers the first formal stability and convergence bounds for predictive coding, linking it to quasi-Newton methods and clarifying its relation to other learning algorithms.

Findings

Predictive coding is Lyapunov stable under mild conditions.

PC updates approximate quasi-Newton methods, enhancing stability and convergence.

PC is closer to quasi-Newton updates than target propagation, with implications for robustness.

Abstract

Energy-based learning algorithms, such as predictive coding (PC), have garnered significant attention in the machine learning community due to their theoretical properties, such as local operations and biologically plausible mechanisms for error correction. In this work, we rigorously analyze the stability, robustness, and convergence of PC through the lens of dynamical systems theory. We show that, first, PC is Lyapunov stable under mild assumptions on its loss and residual energy functions, which implies intrinsic robustness to small random perturbations due to its well-defined energy-minimizing dynamics. Second, we formally establish that the PC updates approximate quasi-Newton methods by incorporating higher-order curvature information, which makes them more stable and able to converge with fewer iterations compared to models trained via backpropagation (BP). Furthermore, using this dynamical framework, we provide new theoretical bounds on the similarity between PC and other algorithms, i.e., BP and target propagation (TP), by precisely characterizing the role of higher-order derivatives. These bounds, derived through detailed analysis of the Hessian structures, show that PC is significantly closer to quasi-Newton updates than TP, providing a deeper understanding of the stability and efficiency of PC compared to conventional learning methods.