Revisiting Sample Size Determination in Natural Language Understanding

TL;DR

This paper investigates methods for estimating the optimal sample size in NLP tasks, proposing a simple approach to predict maximum model performance early in data annotation, thereby aiding resource-efficient model development.

Contribution

It introduces a novel, effective method to predict the upper bound of model performance from limited data, enhancing data annotation strategies in NLP.

Findings

Accurately predicts maximum model performance within 0.9% MAE using only 10% data.

Demonstrates effectiveness across four language understanding tasks.

Provides a practical tool for data quality and sample size estimation.

Abstract

Knowing exactly how many data points need to be labeled to achieve a certain model performance is a hugely beneficial step towards reducing the overall budgets for annotation. It pertains to both active learning and traditional data annotation, and is particularly beneficial for low resource scenarios. Nevertheless, it remains a largely under-explored area of research in NLP. We therefore explored various techniques for estimating the training sample size necessary to achieve a targeted performance value. We derived a simple yet effective approach to predict the maximum achievable model performance based on small amount of training samples - which serves as an early indicator during data annotation for data quality and sample size determination. We performed ablation studies on four language understanding tasks, and showed that the proposed approach allows us to forecast model performance within a small margin of mean absolute error (~ 0.9%) with only 10% data.