Lecture notes: From Gaussian processes to feature learning

TL;DR

These lecture notes explore the Bayesian inference framework for deep and recurrent neural networks, connecting Gaussian processes, feature learning, and kernel methods to understand collective network behavior.

Contribution

They develop a comprehensive theoretical foundation linking Gaussian processes, Bayesian inference, and feature learning in deep neural networks.

Findings

Connection between Gaussian processes and neural network behavior

Extension of lazy-learning to feature learning via adaptive kernels

Insights into kernel rescaling and its relation to neural network training

Abstract

These lecture notes develop the theory of learning in deep and recurrent neuronal networks from the point of view of Bayesian inference. The aim is to enable the reader to understand typical computations found in the literature in this field. Initial chapters develop the theoretical tools, such as probabilities, moment and cumulant-generating functions, and some notions of large deviation theory, as far as they are needed to understand collective network behavior with large numbers of parameters. The main part of the notes derives the theory of Bayesian inference for deep and recurrent networks, starting with the neural network Gaussian process (lazy-learning) limit, which is subsequently extended to study feature learning from the point of view of adaptive kernels. The notes also expose the link between the adaptive kernel approach and approaches of kernel rescaling.