Critical Learning Periods for Multisensory Integration in Deep Networks

TL;DR

This paper demonstrates that neural networks require correlated multisensory signals during early training to develop effective integration skills, highlighting the importance of critical learning periods and complex early dynamics in both artificial and biological systems.

Contribution

It reveals that critical periods are caused by unstable early dynamics, even in deep linear networks, and introduces a source sensitivity measure to analyze internal representations and resilience.

Findings

Critical periods depend on early transient dynamics.

Deep linear networks also exhibit critical periods for multisource integration.

Cross-source reconstruction improves robustness to critical periods.

Abstract

We show that the ability of a neural network to integrate information from diverse sources hinges critically on being exposed to properly correlated signals during the early phases of training. Interfering with the learning process during this initial stage can permanently impair the development of a skill, both in artificial and biological systems where the phenomenon is known as a critical learning period. We show that critical periods arise from the complex and unstable early transient dynamics, which are decisive of final performance of the trained system and their learned representations. This evidence challenges the view, engendered by analysis of wide and shallow networks, that early learning dynamics of neural networks are simple, akin to those of a linear model. Indeed, we show that even deep linear networks exhibit critical learning periods for multi-source integration, while shallow networks do not. To better understand how the internal representations change according to disturbances or sensory deficits, we introduce a new measure of source sensitivity, which allows us to track the inhibition and integration of sources during training. Our analysis of inhibition suggests cross-source reconstruction as a natural auxiliary training objective, and indeed we show that architectures trained with cross-sensor reconstruction objectives are remarkably more resilient to critical periods. Our findings suggest that the recent success in self-supervised multi-modal training compared to previous supervised efforts may be in part due to more robust learning dynamics and not solely due to better architectures and/or more data.