Stochastic Weight Matrix-based Regularization Methods for Deep Neural Networks

TL;DR

This paper introduces two novel regularization techniques for deep neural networks based on weight matrix modifications, demonstrating improved performance and entropy on various datasets and tasks.

Contribution

The paper proposes two new regularization methods, Weight Reinitialization and Weight Shuffling, based on weight matrix modifications, with demonstrated effectiveness across multiple datasets.

Findings

Improved performance on MNIST, CIFAR-10, and JSB Chorales datasets.

Enhanced entropy and robustness of neural networks.

Methods applicable to time series modeling tasks.

Abstract

The aim of this paper is to introduce two widely applicable regularization methods based on the direct modification of weight matrices. The first method, Weight Reinitialization, utilizes a simplified Bayesian assumption with partially resetting a sparse subset of the parameters. The second one, Weight Shuffling, introduces an entropy- and weight distribution-invariant non-white noise to the parameters. The latter can also be interpreted as an ensemble approach. The proposed methods are evaluated on benchmark datasets, such as MNIST, CIFAR-10 or the JSB Chorales database, and also on time series modeling tasks. We report gains both regarding performance and entropy of the analyzed networks. We also made our code available as a GitHub repository (https://github.com/rpatrik96/lod-wmm-2019).