A Theory of Feature Learning

TL;DR

This paper develops a theoretical framework for feature learning, explaining when features can be learned unsupervised and how to evaluate their quality using rate-distortion theory, advancing understanding of deep learning methods.

Contribution

It introduces a formal theoretical model for feature learning and proposes criteria to assess feature quality, filling a gap in the theoretical understanding of deep learning.

Findings

Characterizes conditions for unsupervised feature learning

Provides a method to evaluate feature quality

Links feature learning to rate-distortion theory

Abstract

Feature Learning aims to extract relevant information contained in data sets in an automated fashion. It is driving force behind the current deep learning trend, a set of methods that have had widespread empirical success. What is lacking is a theoretical understanding of different feature learning schemes. This work provides a theoretical framework for feature learning and then characterizes when features can be learnt in an unsupervised fashion. We also provide means to judge the quality of features via rate-distortion theory and its generalizations.