DL-Reg: A Deep Learning Regularization Technique using Linear Regression

TL;DR

DL-Reg is a novel regularization technique for deep neural networks that enforces near-linearity to prevent overfitting, especially effective on small datasets, outperforming existing methods.

Contribution

The paper introduces DL-Reg, a new regularization method that explicitly enforces linearity in deep networks to improve generalization and reduce overfitting.

Findings

DL-Reg significantly outperforms existing regularization techniques.

DL-Reg improves deep network performance on small datasets.

Experimental results validate the effectiveness of the proposed method.

Abstract

Regularization plays a vital role in the context of deep learning by preventing deep neural networks from the danger of overfitting. This paper proposes a novel deep learning regularization method named as DL-Reg, which carefully reduces the nonlinearity of deep networks to a certain extent by explicitly enforcing the network to behave as much linear as possible. The key idea is to add a linear constraint to the objective function of the deep neural networks, which is simply the error of a linear mapping from the inputs to the outputs of the model. More precisely, the proposed DL-Reg carefully forces the network to behave in a linear manner. This linear constraint, which is further adjusted by a regularization factor, prevents the network from the risk of overfitting. The performance of DL-Reg is evaluated by training state-of-the-art deep network models on several benchmark datasets. The experimental results show that the proposed regularization method: 1) gives major improvements over the existing regularization techniques, and 2) significantly improves the performance of deep neural networks, especially in the case of small-sized training datasets.