Formalizing Generalization and Robustness of Neural Networks to Weight Perturbations

TL;DR

This paper analyzes how weight perturbations affect neural network generalization and robustness, offering a new theoretical loss function and empirical validation to improve model reliability under such perturbations.

Contribution

It provides the first comprehensive analysis of neural network robustness to weight perturbations and introduces a novel loss function for training more resilient models.

Findings

Theoretical insights into class margin robustness under weight perturbations

A new loss function improves neural network robustness and generalization

Empirical results validate the theoretical analysis and effectiveness of the proposed method

Abstract

Studying the sensitivity of weight perturbation in neural networks and its impacts on model performance, including generalization and robustness, is an active research topic due to its implications on a wide range of machine learning tasks such as model compression, generalization gap assessment, and adversarial attacks. In this paper, we provide the first integral study and analysis for feed-forward neural networks in terms of the robustness in pairwise class margin and its generalization behavior under weight perturbation. We further design a new theory-driven loss function for training generalizable and robust neural networks against weight perturbations. Empirical experiments are conducted to validate our theoretical analysis. Our results offer fundamental insights for characterizing the generalization and robustness of neural networks against weight perturbations.