CoDA21: Evaluating Language Understanding Capabilities of NLP Models With Context-Definition Alignment

TL;DR

CoDA21 is a new benchmark designed to evaluate the deep language understanding of pretrained models by testing their ability to align definitions with contexts without seeing the words, revealing gaps in current models.

Contribution

The paper introduces CoDA21, a novel challenging benchmark for assessing natural language understanding in PLMs through context-definition alignment tasks.

Findings

Large gap between human and model performance

CoDA21 captures aspects of NLU not covered by existing benchmarks

Requires complex inference and world knowledge

Abstract

Pretrained language models (PLMs) have achieved superhuman performance on many benchmarks, creating a need for harder tasks. We introduce CoDA21 (Context Definition Alignment), a challenging benchmark that measures natural language understanding (NLU) capabilities of PLMs: Given a definition and a context each for k words, but not the words themselves, the task is to align the k definitions with the k contexts. CoDA21 requires a deep understanding of contexts and definitions, including complex inference and world knowledge. We find that there is a large gap between human and PLM performance, suggesting that CoDA21 measures an aspect of NLU that is not sufficiently covered in existing benchmarks.