Robust Neural Network Classification via Double Regularization

TL;DR

This paper introduces DRFit, a double regularization method for neural networks that enhances robustness against mislabeled data and overfitting, supported by theoretical analysis and experiments on MNIST and CIFAR-10.

Contribution

The paper proposes a novel double regularization approach for neural networks that improves robustness to label noise and overfitting, with theoretical and empirical validation.

Findings

DRFit effectively reduces overfitting in the presence of mislabeled data.

DRFit accurately identifies mislabeled data points.

The method improves generalization and label trustworthiness in neural network classification.

Abstract

The presence of mislabeled observations in data is a notoriously challenging problem in statistics and machine learning, associated with poor generalization properties for both traditional classifiers and, perhaps even more so, flexible classifiers like neural networks. Here we propose a novel double regularization of the neural network training loss that combines a penalty on the complexity of the classification model and an optimal reweighting of training observations. The combined penalties result in improved generalization properties and strong robustness against overfitting in different settings of mislabeled training data and also against variation in initial parameter values when training. We provide a theoretical justification for our proposed method derived for a simple case of logistic regression. We demonstrate the double regularization model, here denoted by DRFit, for neural net classification of (i) MNIST and (ii) CIFAR-10, in both cases with simulated mislabeling. We also illustrate that DRFit identifies mislabeled data points with very good precision. This provides strong support for DRFit as a practical of-the-shelf classifier, since, without any sacrifice in performance, we get a classifier that simultaneously reduces overfitting against mislabeling and gives an accurate measure of the trustworthiness of the labels.