The role of a layer in deep neural networks: a Gaussian Process perspective

TL;DR

This paper establishes a novel connection between Gaussian Processes and SGD-trained deep neural networks, leading to explicit layer-wise loss functions that enhance understanding and optimization of internal representations.

Contribution

It introduces Deep Gaussian Layer-wise loss functions (DGLs) derived from a Gaussian Process correspondence, enabling layer-wise optimization and analysis of deep neural networks.

Findings

DGLs are explicit and competitive in accuracy.

The approach offers a new analytic perspective on internal representations.

Provides a foundation for layer-wise optimization in deep learning.

Abstract

A fundamental question in deep learning concerns the role played by individual layers in a deep neural network (DNN) and the transferable properties of the data representations which they learn. To the extent that layers have clear roles, one should be able to optimize them separately using layer-wise loss functions. Such loss functions would describe what is the set of good data representations at each depth of the network and provide a target for layer-wise greedy optimization (LEGO). Here we derive a novel correspondence between Gaussian Processes and SGD trained deep neural networks. Leveraging this correspondence, we derive the Deep Gaussian Layer-wise loss functions (DGLs) which, we believe, are the first supervised layer-wise loss functions which are both explicit and competitive in terms of accuracy. Being highly structured and symmetric, the DGLs provide a promising analytic route to understanding the internal representations generated by DNNs.