Avoiding pathologies in very deep networks

TL;DR

This paper investigates the limitations of deep Gaussian processes and proposes alternative architectures to prevent capacity collapse, enhancing the understanding of deep network regularization and design.

Contribution

It identifies a pathology in standard deep Gaussian processes and introduces an architecture that maintains representational capacity across many layers.

Findings

Standard architectures tend to lose degrees of freedom with depth.

Proposed architecture avoids capacity collapse in deep networks.

Analyzed deep covariance functions and dropout effects on Gaussian processes.

Abstract

Choosing appropriate architectures and regularization strategies for deep networks is crucial to good predictive performance. To shed light on this problem, we analyze the analogous problem of constructing useful priors on compositions of functions. Specifically, we study the deep Gaussian process, a type of infinitely-wide, deep neural network. We show that in standard architectures, the representational capacity of the network tends to capture fewer degrees of freedom as the number of layers increases, retaining only a single degree of freedom in the limit. We propose an alternate network architecture which does not suffer from this pathology. We also examine deep covariance functions, obtained by composing infinitely many feature transforms. Lastly, we characterize the class of models obtained by performing dropout on Gaussian processes.