Four Quadrants of Difficulty: A Simple Categorisation and its Limits

TL;DR

This paper introduces a four-quadrant categorisation of difficulty signals in NLP curriculum learning, revealing that task-agnostic features are largely independent and emphasizing the importance of task-dependent difficulty estimators for better model training.

Contribution

It proposes a novel four-quadrant framework for categorising difficulty signals and systematically analyses their interactions, challenging existing assumptions in curriculum learning.

Findings

Task-agnostic features behave independently from task-dependent ones.

Only task-dependent features align with model difficulty.

Lightweight, task-dependent difficulty estimators are needed for improved training.

Abstract

Curriculum Learning (CL) aims to improve the outcome of model training by estimating the difficulty of samples and scheduling them accordingly. In NLP, difficulty is commonly approximated using task-agnostic linguistic heuristics or human intuition, implicitly assuming that these signals correlate with what neural models find difficult to learn. We propose a four-quadrant categorisation of difficulty signals -- human vs. model and task-agnostic vs. task-dependent -- and systematically analyse their interactions on a natural language understanding dataset. We find that task-agnostic features behave largely independently and that only task-dependent features align. These findings challenge common CL intuitions and highlight the need for lightweight, task-dependent difficulty estimators that better reflect model learning behaviour.