Understanding Sample Efficiency in Predictive Coding

TL;DR

This paper investigates the efficiency of Predictive Coding (PC) compared to Backpropagation (BP) in neural networks, introducing a metric called target alignment to quantify learning effectiveness and providing theoretical and empirical insights.

Contribution

It introduces target alignment as a metric for learning efficiency, derives analytical expressions for it in linear networks, and demonstrates PC's superior efficiency over BP especially in certain network configurations.

Findings

PC achieves higher target alignment than BP, especially in deep, narrow, and pre-trained networks.

Analytical expressions for target alignment are derived and validated.

Benefits of PC persist in practice even when assumptions are violated.

Abstract

Predictive Coding (PC) is an influential account of cortical learning. Much of recent work has focused on comparing PC to Backpropagation (BP) to find whether PC offers any advantages. Small scale experiments show that PC enables learning that is more sample efficient and effective in many contexts, though a thorough theoretical understanding of the phenomena remains elusive. To address this, we quantify the efficiency of learning in BP and PC through a metric called ``target alignment'', which measures how closely the change in the output of the network is aligned to the output prediction error. We then derive and empirically validate analytical expressions for target alignment in Deep Linear Networks. We show that learning in PC is more efficient than BP, which is especially pronounced in deep, narrow and pre-trained networks. We also derive exact conditions for guaranteed optimal target alignment in PC and validate our findings through experiments. We study full training trajectories of linear and non-linear models, and find the predicted benefits of PC persist in practice even when some assumptions are violated. Overall, this work provides a mechanistic understanding of the higher learning efficiency observed for PC over BP in previous works, and can guide how PC should be parametrised to learn most effectively.