Class-wise Activation Unravelling the Engima of Deep Double Descent

TL;DR

This paper investigates the double descent phenomenon in deep learning by analyzing class activation patterns and model complexity, providing empirical insights into over-parameterization and overfitting.

Contribution

It introduces class-activation matrices and a methodology to estimate effective function complexity, offering new understanding of double descent in deep neural networks.

Findings

Over-parameterized models show more distinct class patterns.

Models exhibit overfitting when interpolating noisy labels.

Empirical evidence supports some hypotheses while challenging others.

Abstract

Double descent presents a counter-intuitive aspect within the machine learning domain, and researchers have observed its manifestation in various models and tasks. While some theoretical explanations have been proposed for this phenomenon in specific contexts, an accepted theory for its occurring mechanism in deep learning remains yet to be established. In this study, we revisited the phenomenon of double descent and discussed the conditions of its occurrence. This paper introduces the concept of class-activation matrices and a methodology for estimating the effective complexity of functions, on which we unveil that over-parameterized models exhibit more distinct and simpler class patterns in hidden activations compared to under-parameterized ones. We further looked into the interpolation of noisy labelled data among clean representations and demonstrated overfitting w.r.t. expressive capacity. By comprehensively analysing hypotheses and presenting corresponding empirical evidence that either validates or contradicts these hypotheses, we aim to provide fresh insights into the phenomenon of double descent and benign over-parameterization and facilitate future explorations. By comprehensively studying different hypotheses and the corresponding empirical evidence either supports or challenges these hypotheses, our goal is to offer new insights into the phenomena of double descent and benign over-parameterization, thereby enabling further explorations in the field. The source code is available at https://github.com/Yufei-Gu-451/sparse-generalization.git.