An analytic theory of generalization dynamics and transfer learning in deep linear networks

TL;DR

This paper develops an analytic theory for understanding how deep linear networks learn and transfer knowledge across tasks, explaining generalization behavior and revealing conditions for effective transfer learning.

Contribution

It provides the first analytic solutions for the dynamics of generalization in deep linear networks, including transfer learning, based on task structure and signal-to-noise ratios.

Findings

Deep networks learn important task features first.

Generalization error early in training depends mainly on task structure.

Transfer learning effectiveness depends on task SNR and feature alignment.

Abstract

Much attention has been devoted recently to the generalization puzzle in deep learning: large, deep networks can generalize well, but existing theories bounding generalization error are exceedingly loose, and thus cannot explain this striking performance. Furthermore, a major hope is that knowledge may transfer across tasks, so that multi-task learning can improve generalization on individual tasks. However we lack analytic theories that can quantitatively predict how the degree of knowledge transfer depends on the relationship between the tasks. We develop an analytic theory of the nonlinear dynamics of generalization in deep linear networks, both within and across tasks. In particular, our theory provides analytic solutions to the training and testing error of deep networks as a function of training time, number of examples, network size and initialization, and the task structure and SNR. Our theory reveals that deep networks progressively learn the most important task structure first, so that generalization error at the early stopping time primarily depends on task structure and is independent of network size. This suggests any tight bound on generalization error must take into account task structure, and explains observations about real data being learned faster than random data. Intriguingly our theory also reveals the existence of a learning algorithm that proveably out-performs neural network training through gradient descent. Finally, for transfer learning, our theory reveals that knowledge transfer depends sensitively, but computably, on the SNRs and input feature alignments of pairs of tasks.