Effective dimension of machine learning models

TL;DR

This paper introduces the local effective dimension as a new capacity measure for machine learning models, demonstrating its correlation with generalization error and providing theoretical bounds to better understand model performance.

Contribution

The paper proposes the local effective dimension as a novel capacity measure that better explains generalization in machine learning models compared to existing measures.

Findings

Local effective dimension correlates well with generalization error

Theoretical bounds relate local effective dimension to generalization performance

The measure captures important practical characteristics of models

Abstract

Making statements about the performance of trained models on tasks involving new data is one of the primary goals of machine learning, i.e., to understand the generalization power of a model. Various capacity measures try to capture this ability, but usually fall short in explaining important characteristics of models that we observe in practice. In this study, we propose the local effective dimension as a capacity measure which seems to correlate well with generalization error on standard data sets. Importantly, we prove that the local effective dimension bounds the generalization error and discuss the aptness of this capacity measure for machine learning models.