Critical Learning Periods Emerge Even in Deep Linear Networks

TL;DR

This paper demonstrates that deep linear networks exhibit critical learning periods similar to biological systems, and provides an analytical understanding of their emergence based on network depth, data structure, and task relationships.

Contribution

It introduces the first analytically tractable model showing how critical periods arise in deep linear networks, linking them to network depth, data, and task interactions.

Findings

Critical periods depend on network depth and data structure.

Feature learning involves competition between sources.

Pre-training can impair transfer learning depending on task relationships.

Abstract

Critical learning periods are periods early in development where temporary sensory deficits can have a permanent effect on behavior and learned representations. Despite the radical differences between biological and artificial networks, critical learning periods have been empirically observed in both systems. This suggests that critical periods may be fundamental to learning and not an accident of biology. Yet, why exactly critical periods emerge in deep networks is still an open question, and in particular it is unclear whether the critical periods observed in both systems depend on particular architectural or optimization details. To isolate the key underlying factors, we focus on deep linear network models, and show that, surprisingly, such networks also display much of the behavior seen in biology and artificial networks, while being amenable to analytical treatment. We show that critical periods depend on the depth of the model and structure of the data distribution. We also show analytically and in simulations that the learning of features is tied to competition between sources. Finally, we extend our analysis to multi-task learning to show that pre-training on certain tasks can damage the transfer performance on new tasks, and show how this depends on the relationship between tasks and the duration of the pre-training stage. To the best of our knowledge, our work provides the first analytically tractable model that sheds light into why critical learning periods emerge in biological and artificial networks.