Dynamic Data Selection for Curriculum Learning via Ability Estimation

TL;DR

This paper introduces DDaCLAE, a novel curriculum learning strategy that dynamically estimates model ability to select training data, outperforming heuristic-based methods on GLUE tasks.

Contribution

It proposes replacing heuristic difficulty estimates with learned difficulty parameters and introduces a dynamic data selection method based on ability estimation.

Findings

Models with learned difficulty outperform heuristic-based curricula.

Dynamic ability-based data selection improves performance on GLUE tasks.

The method adapts training data to model ability at each epoch.

Abstract

Curriculum learning methods typically rely on heuristics to estimate the difficulty of training examples or the ability of the model. In this work, we propose replacing difficulty heuristics with learned difficulty parameters. We also propose Dynamic Data selection for Curriculum Learning via Ability Estimation (DDaCLAE), a strategy that probes model ability at each training epoch to select the best training examples at that point. We show that models using learned difficulty and/or ability outperform heuristic-based curriculum learning models on the GLUE classification tasks.