SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

TL;DR

SuperGLUE is a new, more challenging benchmark for evaluating general-purpose language understanding systems, featuring harder tasks, a software toolkit, and a public leaderboard to advance research beyond current models.

Contribution

It introduces SuperGLUE, a benchmark with more difficult tasks and tools to push the limits of language understanding models beyond existing performance levels.

Findings

Performance on GLUE surpasses non-expert human levels.

SuperGLUE provides a more challenging set of tasks.

Benchmark and toolkit are publicly available.

Abstract

In the last year, new models and methods for pretraining and transfer learning have driven striking performance improvements across a range of language understanding tasks. The GLUE benchmark, introduced a little over one year ago, offers a single-number metric that summarizes progress on a diverse set of such tasks, but performance on the benchmark has recently surpassed the level of non-expert humans, suggesting limited headroom for further research. In this paper we present SuperGLUE, a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, a software toolkit, and a public leaderboard. SuperGLUE is available at super.gluebenchmark.com.