Boundary between noise and information applied to filtering neural network weight matrices

TL;DR

This paper introduces a noise filtering algorithm for neural network weight matrices based on a spectral boundary between randomness and learned information, improving generalization in noisy label scenarios.

Contribution

The paper proposes a novel spectral boundary-based noise filtering method for neural network weights, enhancing performance when training data contains label noise.

Findings

Filtering improves generalization performance in noisy label training.

Singular value spectrum reveals a boundary between noise and information.

The method effectively reduces the impact of noise on learned weights.

Abstract

Deep neural networks have been successfully applied to a broad range of problems where overparametrization yields weight matrices which are partially random. A comparison of weight matrix singular vectors to the Porter-Thomas distribution suggests that there is a boundary between randomness and learned information in the singular value spectrum. Inspired by this finding, we introduce an algorithm for noise filtering, which both removes small singular values and reduces the magnitude of large singular values to counteract the effect of level repulsion between the noise and the information part of the spectrum. For networks trained in the presence of label noise, we indeed find that the generalization performance improves significantly due to noise filtering.