HuCurl: Human-induced Curriculum Discovery

TL;DR

This paper presents HuCurl, a framework for discovering effective curricula in machine learning by analyzing sample difficulty, revealing non-monotonic curricula and outperforming existing methods across NLP tasks.

Contribution

HuCurl introduces a novel curriculum discovery framework that identifies non-monotonic curricula and demonstrates improved performance over traditional approaches.

Findings

Discovered curricula are often non-monotonic.

Easy-to-hard or hard-to-easy curricula may underperform.

Curricula for smaller datasets/models generalize well to larger ones.

Abstract

We introduce the problem of curriculum discovery and describe a curriculum learning framework capable of discovering effective curricula in a curriculum space based on prior knowledge about sample difficulty. Using annotation entropy and loss as measures of difficulty, we show that (i): the top-performing discovered curricula for a given model and dataset are often non-monotonic as opposed to monotonic curricula in existing literature, (ii): the prevailing easy-to-hard or hard-to-easy transition curricula are often at the risk of underperforming, and (iii): the curricula discovered for smaller datasets and models perform well on larger datasets and models respectively. The proposed framework encompasses some of the existing curriculum learning approaches and can discover curricula that outperform them across several NLP tasks.