Predicting Generalization in Deep Learning via Local Measures of Distortion

TL;DR

This paper proposes layer-wise complexity measures based on vector quantization techniques like PCA, GMMs, and SVMs to predict generalization in deep learning, demonstrating their effectiveness in correlating with performance.

Contribution

It introduces simple, inexpensive complexity measures applied to deep features that effectively predict generalization, bridging approximation theory and deep learning.

Findings

Complexity measures correlate well with generalization performance.

Layer-wise vector quantization captures essential complexity information.

Results discussed in the 2020 NeurIPS PGDL challenge.

Abstract

We study generalization in deep learning by appealing to complexity measures originally developed in approximation and information theory. While these concepts are challenged by the high-dimensional and data-defined nature of deep learning, we show that simple vector quantization approaches such as PCA, GMMs, and SVMs capture their spirit when applied layer-wise to deep extracted features giving rise to relatively inexpensive complexity measures that correlate well with generalization performance. We discuss our results in 2020 NeurIPS PGDL challenge.