Statistical Mechanics of Learning in the Presence of Outliers

TL;DR

This paper uses statistical mechanics to analyze how outliers affect supervised learning, comparing algorithms that select informative examples either softly or hardly, revealing phase transitions in estimation errors as outliers increase.

Contribution

It introduces a statistical mechanics framework to study outlier effects and compares soft and hard selection algorithms in classification tasks.

Findings

Estimation errors exhibit a first order phase transition with increasing outliers.

Soft and hard selection algorithms behave differently under high outlier fractions.

Outlier presence significantly impacts learning performance and error rates.

Abstract

Using methods of statistical mechanics, we analyse the effect of outliers on the supervised learning of a classification problem. The learning strategy aims at selecting informative examples and discarding outliers. We compare two algorithms which perform the selection either in a soft or a hard way. When the fraction of outliers grows large, the estimation errors undergo a first order phase transition.