ePC: Fast and Deep Predictive Coding for Digital Hardware

TL;DR

This paper introduces ePC, a reparameterized form of Predictive Coding that overcomes efficiency issues in digital simulation, enabling faster training of deep neural networks with performance comparable to backpropagation.

Contribution

It reformulates Predictive Coding into ePC, eliminating signal decay and significantly improving computational speed for deep architectures.

Findings

ePC matches backpropagation performance on multiple datasets

ePC runs orders of magnitude faster than traditional sPC

Theoretical analysis clarifies PC dynamics and scaling potential

Abstract

Predictive Coding (PC) offers a brain-inspired alternative to backpropagation for neural network training, described as a physical system minimizing its internal energy. However, in practice, PC is predominantly digitally simulated, requiring excessive amounts of compute while struggling to scale to deeper architectures. This paper reformulates PC to overcome this hardware-algorithm mismatch. First, we uncover how the canonical state-based formulation of PC (sPC) is, by design, deeply inefficient in digital simulation, inevitably resulting in exponential signal decay that stalls the entire minimization process. Then, to overcome this fundamental limitation, we introduce error-based PC (ePC), a novel reparameterization of PC which does not suffer from signal decay. Though no longer biologically plausible, ePC numerically computes exact PC weights gradients and runs orders of magnitude faster than sPC. Experiments across multiple architectures and datasets demonstrate that ePC matches backpropagation's performance even for deeper models where sPC struggles. Besides practical improvements, our work provides theoretical insight into PC dynamics and establishes a foundation for scaling PC-based learning to deeper architectures on digital hardware and beyond.