A picture of the space of typical learnable tasks

TL;DR

This paper uses information geometry to analyze how deep networks learn representations across various tasks, revealing low-dimensional task manifolds and similarities in learning trajectories across different methods.

Contribution

It introduces geometric techniques to characterize the structure of the task space and compares learning dynamics across supervised, meta-, semi-supervised, and contrastive methods.

Findings

The task manifold is effectively low-dimensional.

Supervised learning on one task benefits other tasks, especially with diverse classes.

The task space structure aligns with the Wordnet phylogenetic tree.

Abstract

We develop information geometric techniques to understand the representations learned by deep networks when they are trained on different tasks using supervised, meta-, semi-supervised and contrastive learning. We shed light on the following phenomena that relate to the structure of the space of tasks: (1) the manifold of probabilistic models trained on different tasks using different representation learning methods is effectively low-dimensional; (2) supervised learning on one task results in a surprising amount of progress even on seemingly dissimilar tasks; progress on other tasks is larger if the training task has diverse classes; (3) the structure of the space of tasks indicated by our analysis is consistent with parts of the Wordnet phylogenetic tree; (4) episodic meta-learning algorithms and supervised learning traverse different trajectories during training but they fit similar models eventually; (5) contrastive and semi-supervised learning methods traverse trajectories similar to those of supervised learning. We use classification tasks constructed from the CIFAR-10 and Imagenet datasets to study these phenomena.