Predictive Coding, Variational Autoencoders, and Biological Connections

TL;DR

This paper explores the shared mathematical foundations of predictive coding in neuroscience and variational autoencoders in machine learning, proposing connections that could foster interdisciplinary advances.

Contribution

It identifies the common framework between predictive coding and variational autoencoders and suggests biological analogies for neural structures and functions.

Findings

Cortical pyramidal dendrites may be analogous to deep networks.

Lateral inhibition could correspond to normalizing flows.

Bridging neuroscience and machine learning through shared mathematical models.

Abstract

This paper reviews predictive coding, from theoretical neuroscience, and variational autoencoders, from machine learning, identifying the common origin and mathematical framework underlying both areas. As each area is prominent within its respective field, more firmly connecting these areas could prove useful in the dialogue between neuroscience and machine learning. After reviewing each area, we discuss two possible correspondences implied by this perspective: cortical pyramidal dendrites as analogous to (non-linear) deep networks and lateral inhibition as analogous to normalizing flows. These connections may provide new directions for further investigations in each field.