A picture of the space of typical learnable tasks
Rahul Ramesh, Jialin Mao, Itay Griniasty, Rubing Yang, Han Kheng Teoh,, Mark Transtrum, James P. Sethna, Pratik Chaudhari

TL;DR
This paper uses information geometry to analyze how deep networks learn representations across various tasks, revealing low-dimensional task manifolds and similarities in learning trajectories across different methods.
Contribution
It introduces geometric techniques to characterize the structure of the task space and compares learning dynamics across supervised, meta-, semi-supervised, and contrastive methods.
Findings
The task manifold is effectively low-dimensional.
Supervised learning on one task benefits other tasks, especially with diverse classes.
The task space structure aligns with the Wordnet phylogenetic tree.
Abstract
We develop information geometric techniques to understand the representations learned by deep networks when they are trained on different tasks using supervised, meta-, semi-supervised and contrastive learning. We shed light on the following phenomena that relate to the structure of the space of tasks: (1) the manifold of probabilistic models trained on different tasks using different representation learning methods is effectively low-dimensional; (2) supervised learning on one task results in a surprising amount of progress even on seemingly dissimilar tasks; progress on other tasks is larger if the training task has diverse classes; (3) the structure of the space of tasks indicated by our analysis is consistent with parts of the Wordnet phylogenetic tree; (4) episodic meta-learning algorithms and supervised learning traverse different trajectories during training but they fit similar…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsNeural Networks and Applications · Image Retrieval and Classification Techniques · Machine Learning and Data Classification
MethodsContrastive Learning
