ILDAE: Instance-Level Difficulty Analysis of Evaluation Data

TL;DR

ILDAE introduces a large-scale method for analyzing question difficulty in NLP evaluation data, enabling efficient evaluation, dataset improvement, model selection, and better Out-of-Domain performance estimation.

Contribution

This paper presents the first large-scale analysis of instance difficulty in NLP evaluation data, demonstrating five novel applications and providing difficulty scores for 23 datasets.

Findings

Using 5% of instances selected by ILDAE achieves 0.93 correlation with full dataset evaluation.

Difficulty scores improve Out-of-Domain performance correlation by 5.2%.

ILDAE's methods enable efficient evaluation and dataset refinement.

Abstract

Knowledge of questions' difficulty level helps a teacher in several ways, such as estimating students' potential quickly by asking carefully selected questions and improving quality of examination by modifying trivial and hard questions. Can we extract such benefits of instance difficulty in NLP? To this end, we conduct Instance-Level Difficulty Analysis of Evaluation data (ILDAE) in a large-scale setup of 23 datasets and demonstrate its five novel applications: 1) conducting efficient-yet-accurate evaluations with fewer instances saving computational cost and time, 2) improving quality of existing evaluation datasets by repairing erroneous and trivial instances, 3) selecting the best model based on application requirements, 4) analyzing dataset characteristics for guiding future data creation, 5) estimating Out-of-Domain performance reliably. Comprehensive experiments for these applications result in several interesting findings, such as evaluation using just 5% instances (selected via ILDAE) achieves as high as 0.93 Kendall correlation with evaluation using complete dataset and computing weighted accuracy using difficulty scores leads to 5.2% higher correlation with Out-of-Domain performance. We release the difficulty scores and hope our analyses and findings will bring more attention to this important yet understudied field of leveraging instance difficulty in evaluations.