Tilting the Odds at the Lottery: the Interplay of Overparameterisation and Curricula in Neural Networks

TL;DR

This paper explores how overparameterisation and curriculum learning interact in neural networks, revealing that excessive overparameterisation can diminish the benefits of curriculum strategies in deep learning.

Contribution

It provides an analytical study connecting overparameterisation and curriculum learning, explaining why curricula may be less effective in highly overparameterised neural networks.

Findings

High overparameterisation simplifies the problem

Overparameterisation can limit curriculum benefits

Theoretical explanation for curricula's limited effectiveness in deep learning

Abstract

A wide range of empirical and theoretical works have shown that overparameterisation can amplify the performance of neural networks. According to the lottery ticket hypothesis, overparameterised networks have an increased chance of containing a sub-network that is well-initialised to solve the task at hand. A more parsimonious approach, inspired by animal learning, consists in guiding the learner towards solving the task by curating the order of the examples, i.e. providing a curriculum. However, this learning strategy seems to be hardly beneficial in deep learning applications. In this work, we undertake an analytical study that connects curriculum learning and overparameterisation. In particular, we investigate their interplay in the online learning setting for a 2-layer network in the XOR-like Gaussian Mixture problem. Our results show that a high degree of overparameterisation -- while simplifying the problem -- can limit the benefit from curricula, providing a theoretical account of the ineffectiveness of curricula in deep learning.