An Empirical Comparison of Syllabuses for Curriculum Learning

TL;DR

This paper empirically compares various curriculum learning syllabuses for training LSTM networks, finding that the choice of syllabus affects learning speed but not generalization, with automated approaches performing strongly.

Contribution

It systematically evaluates different syllabuses for curriculum learning, providing insights into their relative performance and guiding future syllabus design.

Findings

Automated curriculum learning (Predictive Gain) performs competitively.

Choice of syllabus has limited impact on generalization.

Task-dependent syllabus effectiveness for learning speed.

Abstract

Syllabuses for curriculum learning have been developed on an ad-hoc, per task basis and little is known about the relative performance of different syllabuses. We identify a number of syllabuses used in the literature. We compare the identified syllabuses based on their effect on the speed of learning and generalization ability of a LSTM network on three sequential learning tasks. We find that the choice of syllabus has limited effect on the generalization ability of a trained network. In terms of speed of learning our results demonstrate that the best syllabus is task dependent but that a recently proposed automated curriculum learning approach - Predictive Gain, performs very competitively against all identified hand-crafted syllabuses. The best performing hand-crafted syllabus which we term Look Back and Forward combines a syllabus which steps through tasks in the order of their difficulty with a uniform distribution over all tasks. Our experimental results provide an empirical basis for the choice of syllabus on a new problem that could benefit from curriculum learning. Additionally, insights derived from our results shed light on how to successfully design new syllabuses.