Accelerated Predictive Coding Networks via Direct Kolen-Pollack Feedback Alignment

TL;DR

This paper introduces DKP-PC, an improved predictive coding algorithm with direct feedback pathways that significantly reduces error propagation delay, enhances scalability, and maintains competitive performance, making it suitable for hardware-efficient neural network training.

Contribution

The paper proposes DKP-PC, a novel predictive coding method with direct feedback connections that eliminate depth-dependent error delay, improving scalability and efficiency over traditional PC.

Findings

DKP-PC reduces error propagation time complexity from O(L) to O(1).

Empirical results show DKP-PC matches or exceeds standard PC performance.

DKP-PC offers improved latency and computational efficiency for hardware implementations.

Abstract

Predictive coding (PC) is a biologically inspired algorithm for training neural networks that relies only on local updates, allowing parallel learning across layers. However, practical implementations face two key limitations: error signals must still propagate from the output to early layers through multiple inference-phase steps, and feedback decays exponentially during this process, leading to vanishing updates in early layers. We propose direct Kolen-Pollack predictive coding (DKP-PC), which simultaneously addresses both feedback delay and exponential decay, yielding a more efficient and scalable variant of PC while preserving update locality. Leveraging direct feedback alignment and direct Kolen-Pollack algorithms, DKP-PC introduces learnable feedback connections from the output layer to all hidden layers, establishing a direct pathway for error transmission. This yields an algorithm that reduces the theoretical error propagation time complexity from O(L), with L being the network depth, to O(1), removing depth-dependent delay in error signals. Moreover, empirical results demonstrate that DKP-PC achieves performance at least comparable to, and often exceeding, that of standard PC, while offering improved latency and computational performance, supporting its potential for custom hardware-efficient implementations.