# On the Limits of Learning Representations with Label-Based Supervision

## TL;DR

This paper investigates the theoretical limits of label-based supervised learning for representation learning, arguing that generative models like GANs have greater potential for learning transferable features.

## Contribution

It provides an information-theoretic analysis showing supervised learning is upper bounded in representation capacity, unlike certain generative models, motivating further research in generative representation learning.

## Key findings

- Supervised learning has an upper bound in representation capacity.
- Generative models, such as GANs, can potentially learn richer representations.
- The analysis offers a theoretical foundation for exploring generative approaches.

## Abstract

Advances in neural network based classifiers have transformed automatic feature learning from a pipe dream of stronger AI to a routine and expected property of practical systems. Since the emergence of AlexNet every winning submission of the ImageNet challenge has employed end-to-end representation learning, and due to the utility of good representations for transfer learning, representation learning has become as an important and distinct task from supervised learning. At present, this distinction is inconsequential, as supervised methods are state-of-the-art in learning transferable representations. But recent work has shown that generative models can also be powerful agents of representation learning. Will the representations learned from these generative methods ever rival the quality of those from their supervised competitors? In this work, we argue in the affirmative, that from an information theoretic perspective, generative models have greater potential for representation learning. Based on several experimentally validated assumptions, we show that supervised learning is upper bounded in its capacity for representation learning in ways that certain generative models, such as Generative Adversarial Networks (GANs) are not. We hope that our analysis will provide a rigorous motivation for further exploration of generative representation learning.