Towards the Training of Deeper Predictive Coding Neural Networks

TL;DR

This paper identifies key issues limiting deep predictive coding networks and proposes novel solutions, including precision-weighted optimization, error balancing, and auxiliary neurons, enabling performance comparable to backpropagation in deep models.

Contribution

The authors introduce new methods to improve training of deep predictive coding networks, addressing error imbalance, energy propagation, and residual effects, thus enhancing their scalability and effectiveness.

Findings

Achieved performance comparable to backpropagation on deep ResNets.

Balanced error distributions improve training stability.

Auxiliary neurons slow energy propagation in residual connections.

Abstract

Predictive coding networks are neural models that perform inference through an iterative energy minimization process, whose operations are local in space and time. While effective in shallow architectures, they suffer significant performance degradation beyond five to seven layers. In this work, we show that this degradation is caused by exponentially imbalanced errors between layers during weight updates, and by predictions from the previous layers not being effective in guiding updates in deeper layers. Furthermore, when training models with skip connections, the energy propagated by the residuals reaches higher layers faster than that propagated by the main pathway, affecting test accuracy. We address the first issue by introducing a novel precision-weighted optimization of latent variables that balances error distributions during the relaxation phase, the second issue by proposing a novel weight update mechanism that reduces error accumulation in deeper layers, and the third one by using auxiliary neurons that slow down the propagation of the energy in the residual connections. Empirically, our methods achieve performance comparable to backpropagation on deep models such as ResNets, opening new possibilities for predictive coding in complex tasks.