VC Theoretical Explanation of Double Descent

TL;DR

This paper explains the double descent phenomenon in neural networks using classical VC-theory, showing it can be understood through VC-generalization bounds and illustrating with empirical results across various classifiers.

Contribution

It provides a VC-theoretical explanation for double descent, connecting it to classical VC bounds and clarifying misconceptions in deep learning.

Findings

Double descent can be explained by VC-generalization bounds.

Empirical results support VC-theoretical modeling across classifiers.

Discussion on misinterpretations of VC theory in deep learning.

Abstract

There has been growing interest in generalization performance of large multilayer neural networks that can be trained to achieve zero training error, while generalizing well on test data. This regime is known as 'second descent' and it appears to contradict the conventional view that optimal model complexity should reflect an optimal balance between underfitting and overfitting, i.e., the bias-variance trade-off. This paper presents a VC-theoretical analysis of double descent and shows that it can be fully explained by classical VC-generalization bounds. We illustrate an application of analytic VC-bounds for modeling double descent for classification, using empirical results for several learning methods, such as SVM, Least Squares, and Multilayer Perceptron classifiers. In addition, we discuss several reasons for the misinterpretation of VC-theoretical results in Deep Learning community.