Are All the Datasets in Benchmark Necessary? A Pilot Study of Dataset Evaluation for Text Classification

TL;DR

This study questions the necessity of all datasets in text classification benchmarks, revealing that some less-used datasets are more discriminative and proposing a method to predict dataset usefulness based on their properties.

Contribution

The paper introduces a systematic analysis of dataset discriminability in benchmarks and proposes a predictive approach for dataset usefulness based on dataset features.

Findings

Some less-used datasets are highly discriminative.

Existing datasets contribute little to system differentiation.

A predictor can estimate dataset discriminability from features.

Abstract

In this paper, we ask the research question of whether all the datasets in the benchmark are necessary. We approach this by first characterizing the distinguishability of datasets when comparing different systems. Experiments on 9 datasets and 36 systems show that several existing benchmark datasets contribute little to discriminating top-scoring systems, while those less used datasets exhibit impressive discriminative power. We further, taking the text classification task as a case study, investigate the possibility of predicting dataset discrimination based on its properties (e.g., average sentence length). Our preliminary experiments promisingly show that given a sufficient number of training experimental records, a meaningful predictor can be learned to estimate dataset discrimination over unseen datasets. We released all datasets with features explored in this work on DataLab: \url{https://datalab.nlpedia.ai}.