Gi and Pal Scores: Deep Neural Network Generalization Statistics

TL;DR

This paper introduces Gi-score and Pal-score, two new statistical measures inspired by income inequality metrics, to evaluate and predict the generalization capabilities of deep neural networks.

Contribution

The paper proposes novel, robust measures for neural network generalization, addressing the lack of theoretical bounds and consistent evaluation metrics.

Findings

Gi-score and Pal-score effectively predict generalization gaps.

The measures are robust to network invariances.

They provide insights into neural network performance.

Abstract

The field of Deep Learning is rich with empirical evidence of human-like performance on a variety of regression, classification, and control tasks. However, despite these successes, the field lacks strong theoretical error bounds and consistent measures of network generalization and learned invariances. In this work, we introduce two new measures, the Gi-score and Pal-score, that capture a deep neural network's generalization capabilities. Inspired by the Gini coefficient and Palma ratio, measures of income inequality, our statistics are robust measures of a network's invariance to perturbations that accurately predict generalization gaps, i.e., the difference between accuracy on training and test sets.