Learned Random Label Predictions as a Neural Network Complexity Metric

TL;DR

This paper introduces a novel neural network complexity metric based on learning random labels, revealing insights into memorization, regularization, and generalization in deep neural networks.

Contribution

It proposes a multi-head architecture to unlearn random labels, using Rademacher complexity as a complexity measure, and introduces a new regularizer to reduce memorization.

Findings

The method serves as an effective complexity metric.

A new regularizer reduces sample memorization.

No observed improvement in generalization despite reduced memorization.

Abstract

We empirically investigate the impact of learning randomly generated labels in parallel to class labels in supervised learning on memorization, model complexity, and generalization in deep neural networks. To this end, we introduce a multi-head network architecture as an extension of standard CNN architectures. Inspired by methods used in fair AI, our approach allows for the unlearning of random labels, preventing the network from memorizing individual samples. Based on the concept of Rademacher complexity, we first use our proposed method as a complexity metric to analyze the effects of common regularization techniques and challenge the traditional understanding of feature extraction and classification in CNNs. Second, we propose a novel regularizer that effectively reduces sample memorization. However, contrary to the predictions of classical statistical learning theory, we do not observe improvements in generalization.