# The Information Complexity of Learning Tasks, their Structure and their Distance

## TL;DR

This paper introduces a novel framework to measure the complexity and distance of learning tasks, accounting for finite data, optimization effects, and enabling better understanding of transfer learning in deep models.

## Contribution

It develops a non-asymptotic, task complexity framework that integrates classical information measures and is applicable to large-scale, real-world deep learning scenarios.

## Key findings

- Defines an asymmetric distance between learning tasks.
- Framework captures the finite nature of training datasets.
- Accounts for the impact of optimization schemes on complexity.

## Abstract

We introduce an asymmetric distance in the space of learning tasks, and a framework to compute their complexity. These concepts are foundational for the practice of transfer learning, whereby a parametric model is pre-trained for a task, and then fine-tuned for another. The framework we develop is non-asymptotic, captures the finite nature of the training dataset, and allows distinguishing learning from memorization. It encompasses, as special cases, classical notions from Kolmogorov complexity, Shannon, and Fisher Information. However, unlike some of those frameworks, it can be applied to large-scale models and real-world datasets. Our framework is the first to measure complexity in a way that accounts for the effect of the optimization scheme, which is critical in Deep Learning.