Faster Predictive Coding Networks via Better Initialization

TL;DR

This paper introduces a novel initialization method for predictive coding networks that significantly reduces training time and improves convergence, bridging the gap between predictive coding and backpropagation.

Contribution

The authors propose a new neuron initialization technique that preserves iterative progress, enhancing efficiency and performance of predictive coding networks.

Findings

Faster convergence in predictive coding networks.

Reduced final test loss in supervised and unsupervised tasks.

Significant improvement in training speed.

Abstract

Research aimed at scaling up neuroscience inspired learning algorithms for neural networks is accelerating. Recently, a key research area has been the study of energy-based learning algorithms such as predictive coding, due to their versatility and mathematical grounding. However, the applicability of such methods is held back by the large computational requirements caused by their iterative nature. In this work, we address this problem by showing that the choice of initialization of the neurons in a predictive coding network matters significantly and can notably reduce the required training times. Consequently, we propose a new initialization technique for predictive coding networks that aims to preserve the iterative progress made on previous training samples. Our approach suggests a promising path toward reconciling the disparities between predictive coding and backpropagation in terms of computational efficiency and final performance. In fact, our experiments demonstrate substantial improvements in convergence speed and final test loss in both supervised and unsupervised settings.