Spectral Norm Regularization for Improving the Generalizability of Deep Learning

TL;DR

This paper introduces spectral norm regularization to improve deep learning models' generalizability by reducing their sensitivity to input perturbations, demonstrating enhanced performance over baseline methods.

Contribution

The paper proposes a novel spectral norm regularization technique that penalizes high spectral norms in neural network weights to enhance generalization.

Findings

Models with spectral norm regularization show better generalization.

Spectral norm regularization reduces sensitivity to input perturbations.

Experimental results outperform baseline methods.

Abstract

We investigate the generalizability of deep learning based on the sensitivity to input perturbation. We hypothesize that the high sensitivity to the perturbation of data degrades the performance on it. To reduce the sensitivity to perturbation, we propose a simple and effective regularization method, referred to as spectral norm regularization, which penalizes the high spectral norm of weight matrices in neural networks. We provide supportive evidence for the abovementioned hypothesis by experimentally confirming that the models trained using spectral norm regularization exhibit better generalizability than other baseline methods.