A Challenging Benchmark for Low-Resource Learning

TL;DR

This paper introduces hardBench, a challenging benchmark with 11 datasets to better evaluate neural networks' robustness in low-resource settings, revealing significant performance gaps and weaknesses.

Contribution

The paper presents hardBench, a new benchmark covering diverse datasets, and provides a theoretical analysis of low-resource learning difficulties, highlighting existing models' limitations.

Findings

Neural networks perform poorly on hardBench, exposing robustness issues.

Pre-trained models do not improve on hardBench despite better traditional benchmarks.

Significant performance gap remains between models and human-level understanding.

Abstract

With promising yet saturated results in high-resource settings, low-resource datasets have gradually become popular benchmarks for evaluating the learning ability of advanced neural networks (e.g., BigBench, superGLUE). Some models even surpass humans according to benchmark test results. However, we find that there exists a set of hard examples in low-resource settings that challenge neural networks but are not well evaluated, which causes over-estimated performance. We first give a theoretical analysis on which factors bring the difficulty of low-resource learning. It then motivate us to propose a challenging benchmark hardBench to better evaluate the learning ability, which covers 11 datasets, including 3 computer vision (CV) datasets and 8 natural language process (NLP) datasets. Experiments on a wide range of models show that neural networks, even pre-trained language models, have sharp performance drops on our benchmark, demonstrating the effectiveness on evaluating the weaknesses of neural networks. On NLP tasks, we surprisingly find that despite better results on traditional low-resource benchmarks, pre-trained networks, does not show performance improvements on our benchmarks. These results demonstrate that there are still a large robustness gap between existing models and human-level performance.