Long-Tail Theory under Gaussian Mixtures

TL;DR

This paper models long-tail data using Gaussian mixtures and shows that nonlinear classifiers outperform linear ones in generalization, especially with longer tails, highlighting the importance of rare examples.

Contribution

It introduces a Gaussian mixture model aligned with Feldman's long tail theory and demonstrates the impact of classifier type and tail length on generalization performance.

Findings

Nonlinear classifiers can surpass linear classifiers in long-tail settings.

The performance gap decreases as the tail shortens.

Experimental validation on synthetic and real data supports the theory.

Abstract

We suggest a simple Gaussian mixture model for data generation that complies with Feldman's long tail theory (2020). We demonstrate that a linear classifier cannot decrease the generalization error below a certain level in the proposed model, whereas a nonlinear classifier with a memorization capacity can. This confirms that for long-tailed distributions, rare training examples must be considered for optimal generalization to new data. Finally, we show that the performance gap between linear and nonlinear models can be lessened as the tail becomes shorter in the subpopulation frequency distribution, as confirmed by experiments on synthetic and real data.