Scale-invariant representation of machine learning

TL;DR

This paper explores how machine learning models naturally develop scale-invariant internal representations characterized by power-law distributions, which reflect data compression and differentiation of outliers, rooted in information theory.

Contribution

It derives the theoretical process explaining the emergence of power-law distributions in machine learning representations, linking data compression and uncertainty.

Findings

Internal codes follow power-law distributions in models.

Scale-invariance relates to maximally uncertain data groupings.

Representation efficiency balances compression and differentiation.

Abstract

The success of machine learning has resulted from its structured representation of data. Similar data have close internal representations as compressed codes for classification or emerged labels for clustering. We observe that the frequency of internal codes or labels follows power laws in both supervised and unsupervised learning models. This scale-invariant distribution implies that machine learning largely compresses frequent typical data, and simultaneously, differentiates many atypical data as outliers. In this study, we derive the process by which these power laws can naturally arise in machine learning. In terms of information theory, the scale-invariant representation corresponds to a maximally uncertain data grouping among possible representations that guarantee a given learning accuracy.