On the Stability of Neural Networks in Deep Learning

TL;DR

This paper explores the stability and robustness of neural networks in deep learning by analyzing sensitivity, proposing Lipschitz constraints, regularization techniques, and randomized smoothing to improve generalization and adversarial robustness.

Contribution

It introduces a unified framework combining Lipschitz continuity, curvature regularization, and randomized smoothing, along with practical methods for enhancing neural network stability.

Findings

Lipschitz networks improve robustness and generalization.

Curvature-based regularization leads to smoother optimization landscapes.

Randomized smoothing enhances decision boundary robustness.

Abstract

Deep learning has achieved remarkable success across a wide range of tasks, but its models often suffer from instability and vulnerability: small changes to the input may drastically affect predictions, while optimization can be hindered by sharp loss landscapes. This thesis addresses these issues through the unifying perspective of sensitivity analysis, which examines how neural networks respond to perturbations at both the input and parameter levels. We study Lipschitz networks as a principled way to constrain sensitivity to input perturbations, thereby improving generalization, adversarial robustness, and training stability. To complement this architectural approach, we introduce regularization techniques based on the curvature of the loss function, promoting smoother optimization landscapes and reducing sensitivity to parameter variations. Randomized smoothing is also explored as a probabilistic method for enhancing robustness at decision boundaries. By combining these perspectives, we develop a unified framework where Lipschitz continuity, randomized smoothing, and curvature regularization interact to address fundamental challenges in stability. The thesis contributes both theoretical analysis and practical methodologies, including efficient spectral norm computation, novel Lipschitz-constrained layers, and improved certification procedures.