Signed Input Regularization

TL;DR

SIGN is a novel regularization technique that modifies input variables through linear transformation to improve model robustness and performance, especially against out-of-distribution samples and corruptions.

Contribution

The paper introduces SIGN, a new input regularization method that de-emphasizes less important variables, enhancing robustness and transferability across models.

Findings

SIGN improves robustness to corruptions and out-of-distribution samples.

SIGN achieves superior classification performance on standard data.

SIGN promotes more compact class representations.

Abstract

Over-parameterized deep models usually over-fit to a given training distribution, which makes them sensitive to small changes and out-of-distribution samples at inference time, leading to low generalization performance. To this end, several model-based and randomized data-dependent regularization methods are applied, such as data augmentation, which prevents a model from memorizing the training distribution. Instead of the random transformation of the input images, we propose SIGN, a new regularization method, which modifies the input variables using a linear transformation by estimating each variable's contribution to the final prediction. Our proposed technique maps the input data to a new manifold where the less important variables are de-emphasized. To test the effectiveness of the proposed idea and compare it with other competing methods, we design several test scenarios, such as classification performance, uncertainty, out-of-distribution, and robustness analyses. We compare the methods using three different datasets and four models. We find that SIGN encourages more compact class representations, which results in the model's robustness to random corruptions and out-of-distribution samples while also simultaneously achieving superior performance on normal data compared to other competing methods. Our experiments also demonstrate the successful transferability of the SIGN samples from one model to another.