SemEval-2013 Task 4: Free Paraphrases of Noun Compounds

TL;DR

This paper presents SemEval-2013 Task 4, which involves paraphrasing English noun compounds and evaluating system performance against human annotations, highlighting the task's difficulty and the challenge of capturing nuanced meanings.

Contribution

It introduces a new paraphrasing task for noun compounds, along with data, evaluation methods, and initial system results, advancing research in semantic paraphrasing.

Findings

All participating systems outperformed the baseline on one measure.

The task proved to be challenging for current systems.

Grouping similar paraphrases improved evaluation reliability.

Abstract

In this paper, we describe SemEval-2013 Task 4: the definition, the data, the evaluation and the results. The task is to capture some of the meaning of English noun compounds via paraphrasing. Given a two-word noun compound, the participating system is asked to produce an explicitly ranked list of its free-form paraphrases. The list is automatically compared and evaluated against a similarly ranked list of paraphrases proposed by human annotators, recruited and managed through Amazon's Mechanical Turk. The comparison of raw paraphrases is sensitive to syntactic and morphological variation. The "gold" ranking is based on the relative popularity of paraphrases among annotators. To make the ranking more reliable, highly similar paraphrases are grouped, so as to downplay superficial differences in syntax and morphology. Three systems participated in the task. They all beat a simple baseline on one of the two evaluation measures, but not on both measures. This shows that the task is difficult.