What is SemEval evaluating? A Systematic Analysis of Evaluation Campaigns in NLP

TL;DR

This paper systematically analyzes SemEval, the main NLP evaluation campaign, to understand the types of tasks, metrics, architectures, and participation patterns that define its focus and evolution over time.

Contribution

It provides the first comprehensive quantitative overview of SemEval's task types, evaluation metrics, and participation trends, revealing what the NLP community emphasizes in evaluations.

Findings

SemEval predominantly features certain task types and metrics.

Participation and citation patterns have evolved over time.

The analysis highlights the core focus areas of SemEval in NLP evaluation.

Abstract

SemEval is the primary venue in the NLP community for the proposal of new challenges and for the systematic empirical evaluation of NLP systems. This paper provides a systematic quantitative analysis of SemEval aiming to evidence the patterns of the contributions behind SemEval. By understanding the distribution of task types, metrics, architectures, participation and citations over time we aim to answer the question on what is being evaluated by SemEval.