Exploring the Function Space of Deep-Learning Machines

TL;DR

This paper investigates the function space of deep-learning models, revealing how entropy decreases with layers and error, and identifying phase transitions, using physics-inspired methods for both sparse and dense architectures.

Contribution

It introduces a physics-inspired framework to analyze the function space of deep networks, highlighting layer-wise convergence and phase transitions in error.

Findings

Entropy decreases with layers approaching the reference function

Large number of layers enhances convergence

Phase transitions occur as error increases

Abstract

The function space of deep-learning machines is investigated by studying growth in the entropy of functions of a given error with respect to a reference function, realized by a deep-learning machine. Using physics-inspired methods we study both sparsely and densely-connected architectures to discover a layer-wise convergence of candidate functions, marked by a corresponding reduction in entropy when approaching the reference function, gain insight into the importance of having a large number of layers, and observe phase transitions as the error increases.