Adding noise to the input of a model trained with a regularized objective

TL;DR

This paper analyzes how adding noise to model inputs influences regularization, focusing on penalizing the Hessian and Jacobian to improve generalization in neural networks.

Contribution

It derives higher order regularization terms from input noise and proposes methods to control regularization strength via penalizing derivatives of the model.

Findings

Penalizing the Hessian improves generalization.

Controlling the Jacobian independently affects regularization.

Theoretical analysis of noise-induced regularization terms.

Abstract

Regularization is a well studied problem in the context of neural networks. It is usually used to improve the generalization performance when the number of input samples is relatively small or heavily contaminated with noise. The regularization of a parametric model can be achieved in different manners some of which are early stopping (Morgan and Bourlard, 1990), weight decay, output smoothing that are used to avoid overfitting during the training of the considered model. From a Bayesian point of view, many regularization techniques correspond to imposing certain prior distributions on model parameters (Krogh and Hertz, 1991). Using Bishop's approximation (Bishop, 1995) of the objective function when a restricted type of noise is added to the input of a parametric function, we derive the higher order terms of the Taylor expansion and analyze the coefficients of the regularization terms induced by the noisy input. In particular we study the effect of penalizing the Hessian of the mapping function with respect to the input in terms of generalization performance. We also show how we can control independently this coefficient by explicitly penalizing the Jacobian of the mapping function on corrupted inputs.