Minimum sharpness: Scale-invariant parameter-robustness of neural networks

TL;DR

This paper introduces Minimum Sharpness, a scale-invariant measure of neural network robustness that correlates with generalization and is computationally efficient compared to existing methods.

Contribution

It proposes a novel scale-invariant sharpness measure for neural networks and provides an efficient technique to compute it, addressing scale-sensitivity issues.

Findings

Minimum Sharpness correlates with neural network generalization.

The method reduces computational costs compared to existing sharpness measures.

The approach is invariant to scale transformations of neural network parameters.

Abstract

Toward achieving robust and defensive neural networks, the robustness against the weight parameters perturbations, i.e., sharpness, attracts attention in recent years (Sun et al., 2020). However, sharpness is known to remain a critical issue, "scale-sensitivity." In this paper, we propose a novel sharpness measure, Minimum Sharpness. It is known that NNs have a specific scale transformation that constitutes equivalent classes where functional properties are completely identical, and at the same time, their sharpness could change unlimitedly. We define our sharpness through a minimization problem over the equivalent NNs being invariant to the scale transformation. We also develop an efficient and exact technique to make the sharpness tractable, which reduces the heavy computational costs involved with Hessian. In the experiment, we observed that our sharpness has a valid correlation with the generalization of NNs and runs with less computational cost than existing sharpness measures.