Revisiting a Pain in the Neck: Semantic Phrase Processing Benchmark for Language Models

TL;DR

LexBench is a new comprehensive benchmark for evaluating language models on semantic phrase processing tasks, revealing insights into model scaling, few-shot learning, and human-level performance.

Contribution

It introduces LexBench, the first framework to compare models on semantic phrase tasks, covering lexical, idiomatic, noun compounds, and verbal constructions.

Findings

Large models outperform smaller ones in most tasks.

Few-shot models lag behind fine-tuned models in semantic relation categorization.

Strong models achieve performance comparable to humans in semantic phrase processing.

Abstract

We introduce LexBench, a comprehensive evaluation suite enabled to test language models (LMs) on ten semantic phrase processing tasks. Unlike prior studies, it is the first work to propose a framework from the comparative perspective to model the general semantic phrase (i.e., lexical collocation) and three fine-grained semantic phrases, including idiomatic expression, noun compound, and verbal construction. Thanks to \ourbenchmark, we assess the performance of 15 LMs across model architectures and parameter scales in classification, extraction, and interpretation tasks. Through the experiments, we first validate the scaling law and find that, as expected, large models excel better than the smaller ones in most tasks. Second, we investigate further through the scaling semantic relation categorization and find that few-shot LMs still lag behind vanilla fine-tuned models in the task. Third, through human evaluation, we find that the performance of strong models is comparable to the human level regarding semantic phrase processing. Our benchmarking findings can serve future research aiming to improve the generic capability of LMs on semantic phrase comprehension. Our source code and data are available at https://github.com/jacklanda/LexBench