On the Strong Correlation Between Model Invariance and Generalization

TL;DR

This paper introduces a new measure called effective invariance (EI) to quantify model invariance without labels and demonstrates a strong linear correlation between invariance and generalization across models and datasets.

Contribution

The paper proposes EI as a label-free invariance measure and provides large-scale empirical evidence linking invariance to generalization performance.

Findings

Invariance and generalization exhibit a strong linear relationship across models.

Model accuracy and invariance are linearly correlated across different test sets.

EI effectively measures invariance without relying on image labels.

Abstract

Generalization and invariance are two essential properties of any machine learning model. Generalization captures a model's ability to classify unseen data while invariance measures consistency of model predictions on transformations of the data. Existing research suggests a positive relationship: a model generalizing well should be invariant to certain visual factors. Building on this qualitative implication we make two contributions. First, we introduce effective invariance (EI), a simple and reasonable measure of model invariance which does not rely on image labels. Given predictions on a test image and its transformed version, EI measures how well the predictions agree and with what level of confidence. Second, using invariance scores computed by EI, we perform large-scale quantitative correlation studies between generalization and invariance, focusing on rotation and grayscale transformations. From a model-centric view, we observe generalization and invariance of different models exhibit a strong linear relationship, on both in-distribution and out-of-distribution datasets. From a dataset-centric view, we find a certain model's accuracy and invariance linearly correlated on different test sets. Apart from these major findings, other minor but interesting insights are also discussed.