Improving Generalization of Deep Neural Networks by Optimum Shifting

TL;DR

This paper introduces a novel 'optimum shifting' method that adjusts neural network parameters from sharp to flat minima to improve generalization, using linear algebra and Neural Collapse theory for efficiency.

Contribution

The paper proposes a new technique for shifting neural network solutions to flatter minima, enhancing generalization without retraining from scratch.

Findings

Improves generalization by shifting to flatter minima.

Effective across various architectures and datasets.

Reduces computational costs with stochastic optimization.

Abstract

Recent studies showed that the generalization of neural networks is correlated with the sharpness of the loss landscape, and flat minima suggests a better generalization ability than sharp minima. In this paper, we propose a novel method called \emph{optimum shifting}, which changes the parameters of a neural network from a sharp minimum to a flatter one while maintaining the same training loss value. Our method is based on the observation that when the input and output of a neural network are fixed, the matrix multiplications within the network can be treated as systems of under-determined linear equations, enabling adjustment of parameters in the solution space, which can be simply accomplished by solving a constrained optimization problem. Furthermore, we introduce a practical stochastic optimum shifting technique utilizing the Neural Collapse theory to reduce computational costs and provide more degrees of freedom for optimum shifting. Extensive experiments (including classification and detection) with various deep neural network architectures on benchmark datasets demonstrate the effectiveness of our method.