Deep Learning with Eigenvalue Decay Regularizer

TL;DR

This paper introduces Eigenvalue Decay as a regularizer for deep neural networks, enabling back-propagation through a soft approximation of the dominant eigenvalue, with implementation in Keras and evaluation on multiple benchmark datasets.

Contribution

It extends Eigenvalue Decay regularization to deep networks and multiclass problems, providing a differentiable approximation suitable for back-propagation.

Findings

Improved regularization performance on benchmark datasets

Effective integration with Keras for deep learning models

Enhanced theoretical understanding of eigenvalue-based regularization

Abstract

This paper extends our previous work on regularization of neural networks using Eigenvalue Decay by employing a soft approximation of the dominant eigenvalue in order to enable the calculation of its derivatives in relation to the synaptic weights, and therefore the application of back-propagation, which is a primary demand for deep learning. Moreover, we extend our previous theoretical analysis to deep neural networks and multiclass classification problems. Our method is implemented as an additional regularizer in Keras, a modular neural networks library written in Python, and evaluated in the benchmark data sets Reuters Newswire Topics Classification, IMDB database for binary sentiment classification, MNIST database of handwritten digits and CIFAR-10 data set for image classification.