ConsistentFeature: A Plug-and-Play Component for Neural Network Regularization

TL;DR

ConsistentFeature is a versatile regularization method that constrains feature differences across data subsets, effectively reducing overfitting and improving neural network performance across various architectures and tasks.

Contribution

It introduces a simple, adaptable regularization technique based on constraining feature differences, applicable to nearly any neural network architecture and task.

Findings

Reduces overfitting effectively

Improves accuracy and reduces validation loss

Promotes normal convergence even in overfitting models

Abstract

Over-parameterized neural network models often lead to significant performance discrepancies between training and test sets, a phenomenon known as overfitting. To address this, researchers have proposed numerous regularization techniques tailored to various tasks and model architectures. In this paper, we introduce a simple perspective on overfitting: models learn different representations in different i.i.d. datasets. Based on this viewpoint, we propose an adaptive method, ConsistentFeature, that regularizes the model by constraining feature differences across random subsets of the same training set. Due to minimal prior assumptions, this approach is applicable to almost any architecture and task. Our experiments show that it effectively reduces overfitting, with low sensitivity to hyperparameters and minimal computational cost. It demonstrates particularly strong memory suppression and promotes normal convergence, even when the model has already started to overfit. Even in the absence of significant overfitting, our method consistently improves accuracy and reduces validation loss.