Unified Regularity Measures for Sample-wise Learning and Generalization

TL;DR

This paper introduces unified measures of sample regularity based on cumulative classification events to analyze and improve learning and generalization in deep neural networks, validated through experiments.

Contribution

It proposes a pair of novel, formulation-consistent sample regularity measures that unify the analysis of memorization and generalization in neural network training.

Findings

The measures effectively quantify sample stability and uncertainty.

They are robust and beneficial for sample selection in training and testing.

Experiments demonstrate improved understanding and potential performance gains.

Abstract

Fundamental machine learning theory shows that different samples contribute unequally both in learning and testing processes. Contemporary studies on DNN imply that such sample difference is rooted on the distribution of intrinsic pattern information, namely sample regularity. Motivated by the recent discovery on network memorization and generalization, we proposed a pair of sample regularity measures for both processes with a formulation-consistent representation. Specifically, cumulative binary training/generalizing loss (CBTL/CBGL), the cumulative number of correct classiffcations of the training/testing sample within training stage, is proposed to quantize the stability in memorization-generalization process; while forgetting/mal-generalizing events, i.e., the mis-classification of previously learned or generalized sample, are utilized to represent the uncertainty of sample regularity with respect to optimization dynamics. Experiments validated the effectiveness and robustness of the proposed approaches for mini-batch SGD optimization. Further applications on training/testing sample selection show the proposed measures sharing the unified computing procedure could benefit for both tasks.