Statistical Mechanics of Learning: A Variational Approach for Real Data

TL;DR

This paper introduces a variational method extending statistical physics techniques to real data, enabling the computation of approximate generalization error estimators solely from training data.

Contribution

It generalizes the statistical physics approach for learning from random data to real-world datasets using a variational framework.

Findings

Validates the variational approach on real data

Provides estimators for generalization errors from training data

Demonstrates relevance and applicability of the method

Abstract

Using a variational technique, we generalize the statistical physics approach of learning from random examples to make it applicable to real data. We demonstrate the validity and relevance of our method by computing approximate estimators for generalization errors that are based on training data alone.