Bio-Inspired Artificial Neural Networks based on Predictive Coding

TL;DR

This paper introduces predictive coding as a biologically plausible alternative to backpropagation for training neural networks, emphasizing local learning rules and connections to Bayesian inference.

Contribution

It provides a comprehensive tutorial on predictive coding, detailing its mathematical formulation, derivation, and practical implementation with Python examples.

Findings

Predictive coding offers a biologically plausible learning mechanism.

Connections between predictive coding, backpropagation, and Kalman filtering are elucidated.

The tutorial includes practical Python code for implementing predictive coding.

Abstract

Backpropagation (BP) of errors is the backbone training algorithm for artificial neural networks (ANNs). It updates network weights through gradient descent to minimize a loss function representing the mismatch between predictions and desired outputs. BP uses the chain rule to propagate the loss gradient backward through the network hierarchy, allowing efficient weight updates. However, this process requires weight updates at every layer to rely on a global error signal generated at the network's output. In contrast, the Hebbian model of synaptic plasticity states that weight updates are local, depending only on the activity of pre- and post-synaptic neurons. This suggests biological brains likely do not implement BP directly. Recently, Predictive Coding (PC) has gained interest as a biologically plausible alternative that updates weights using only local information. Originating from 1950s work on signal compression, PC was later proposed as a model of the visual cortex and formalized under the free energy principle, linking it to Bayesian inference and dynamical systems. PC weight updates rely solely on local information and provide theoretical advantages such as automatic scaling of gradients based on uncertainty. This lecture notes column offers a novel, tutorial-style introduction to PC, focusing on its formulation, derivation, and connections to well-known optimization and signal processing algorithms such as BP and the Kalman Filter (KF). It aims to support existing literature by guiding readers from the mathematical foundations of PC to practical implementation, including Python examples using PyTorch.