LocalDrop: A Hybrid Regularization for Deep Neural Networks

TL;DR

LocalDrop introduces a novel regularization method based on local Rademacher complexity, improving neural network generalization by optimizing keep rate and weight matrices, with extensive experimental validation across models.

Contribution

It proposes a new regularization function for neural networks using local Rademacher complexity, including a two-stage optimization process for hyperparameters.

Findings

Effective in reducing overfitting across different neural network architectures

Outperforms several existing regularization algorithms in experiments

Hyperparameter tuning significantly impacts model performance

Abstract

In neural networks, developing regularization algorithms to settle overfitting is one of the major study areas. We propose a new approach for the regularization of neural networks by the local Rademacher complexity called LocalDrop. A new regularization function for both fully-connected networks (FCNs) and convolutional neural networks (CNNs), including drop rates and weight matrices, has been developed based on the proposed upper bound of the local Rademacher complexity by the strict mathematical deduction. The analyses of dropout in FCNs and DropBlock in CNNs with keep rate matrices in different layers are also included in the complexity analyses. With the new regularization function, we establish a two-stage procedure to obtain the optimal keep rate matrix and weight matrix to realize the whole training model. Extensive experiments have been conducted to demonstrate the effectiveness of LocalDrop in different models by comparing it with several algorithms and the effects of different hyperparameters on the final performances.