Long-Tailed Learning Requires Feature Learning

TL;DR

This paper introduces a data model inspired by natural data to study how feature learning impacts generalization in long-tailed distributions, emphasizing the necessity of feature learning for successful learning.

Contribution

It provides a theoretical framework and error bounds demonstrating that learning the correct features is essential for good generalization in long-tailed data distributions.

Findings

Feature learning is crucial for success in long-tailed data scenarios.

The paper derives explicit generalization error bounds related to feature learning.

Learning the wrong features leads to significant penalties in generalization.

Abstract

We propose a simple data model inspired from natural data such as text or images, and use it to study the importance of learning features in order to achieve good generalization. Our data model follows a long-tailed distribution in the sense that some rare subcategories have few representatives in the training set. In this context we provide evidence that a learner succeeds if and only if it identifies the correct features, and moreover derive non-asymptotic generalization error bounds that precisely quantify the penalty that one must pay for not learning features.