Do Data-based Curricula Work?

TL;DR

This paper evaluates data-based curriculum learning for large language models like BERT and T5, finding that such curricula rarely outperform random sampling in NLP tasks, questioning their practical benefits.

Contribution

The study systematically assesses various data-based curricula for large models, revealing limited advantages over simple random sampling strategies.

Findings

Curricula based on complexity measures often do not improve performance.

Random sampling performs as well or better than curriculum-based methods.

Extensive experiments across multiple NLP tasks support these conclusions.

Abstract

Current state-of-the-art NLP systems use large neural networks that require lots of computational resources for training. Inspired by human knowledge acquisition, researchers have proposed curriculum learning, - sequencing of tasks (task-based curricula) or ordering and sampling of the datasets (data-based curricula) that facilitate training. This work investigates the benefits of data-based curriculum learning for large modern language models such as BERT and T5. We experiment with various curricula based on a range of complexity measures and different sampling strategies. Extensive experiments on different NLP tasks show that curricula based on various complexity measures rarely has any benefits while random sampling performs either as well or better than curricula.