Investigating Generalization by Controlling Normalized Margin

TL;DR

This paper investigates whether normalized margin causally influences neural network generalization by explicitly controlling it in experiments, revealing that the relationship is context-dependent and not universally causal.

Contribution

It introduces experimental methods to control normalized margin and challenges the assumption of its universal causal role in generalization.

Findings

Normalized margin does not always correlate with generalization.

In standard training, normalized margin closely tracks test performance.

A Gaussian process model may explain the observed behavior.

Abstract

Weight norm $\|w\|$ and margin $\gamma$ participate in learning theory via the normalized margin $\gamma/\|w\|$. Since standard neural net optimizers do not control normalized margin, it is hard to test whether this quantity causally relates to generalization. This paper designs a series of experimental studies that explicitly control normalized margin and thereby tackle two central questions. First: does normalized margin always have a causal effect on generalization? The paper finds that no -- networks can be produced where normalized margin has seemingly no relationship with generalization, counter to the theory of Bartlett et al. (2017). Second: does normalized margin ever have a causal effect on generalization? The paper finds that yes -- in a standard training setup, test performance closely tracks normalized margin. The paper suggests a Gaussian process model as a promising explanation for this behavior.