Bridging Predictive Coding and MDL: A Two-Part Code Framework for Deep Learning

TL;DR

This paper establishes a theoretical connection between predictive coding and the MDL principle in deep learning, providing formal guarantees for generalization and convergence, and positioning PC as a biologically plausible alternative to backpropagation.

Contribution

It introduces the first theoretical framework linking predictive coding with MDL, proving layerwise PC performs block-coordinate descent on the MDL objective, with convergence and generalization guarantees.

Findings

Layerwise PC performs block-coordinate descent on MDL objective.

Repeated PC updates converge to a stationary point.

Provides formal generalization bounds and convergence guarantees.

Abstract

We present the first theoretical framework that connects predictive coding (PC), a biologically inspired local learning rule, with the minimum description length (MDL) principle in deep networks. We prove that layerwise PC performs block-coordinate descent on the MDL two-part code objective, thereby jointly minimizing empirical risk and model complexity. Using Hoeffding's inequality and a prefix-code prior, we derive a novel generalization bound of the form $R(\theta) \le \hat{R}(\theta) + \frac{L(\theta)}{N}$, capturing the tradeoff between fit and compression. We further prove that each PC sweep monotonically decreases the empirical two-part codelength, yielding tighter high-probability risk bounds than unconstrained gradient descent. Finally, we show that repeated PC updates converge to a block-coordinate stationary point, providing an approximate MDL-optimal solution. To our knowledge, this is the first result offering formal generalization and convergence guarantees for PC-trained deep models, positioning PC as a theoretically grounded and biologically plausible alternative to backpropagation.