Regularized deep learning with nonconvex penalties

TL;DR

This paper explores the use of nonconvex penalties for deep neural network regularization, providing theoretical support and empirical evaluation across multiple datasets to improve over traditional convex penalties.

Contribution

It introduces new nonconvex penalties for DNN regularization, with theoretical justifications and comprehensive performance assessments.

Findings

Nonconvex penalties perform well with standard optimization algorithms.

Theoretical guarantees support the use of certain nonconvex penalties.

Empirical results show competitive or improved regularization effects.

Abstract

Regularization methods are often employed in deep learning neural networks (DNNs) to prevent overfitting. For penalty based DNN regularization methods, convex penalties are typically considered because of their optimization guarantees. Recent theoretical work have shown that nonconvex penalties that satisfy certain regularity conditions are also guaranteed to perform well with standard optimization algorithms. In this paper, we examine new and currently existing nonconvex penalties for DNN regularization. We provide theoretical justifications for the new penalties and also assess the performance of all penalties with DNN analyses of seven datasets.