SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation

TL;DR

The paper presents the 2017 SemEval STS shared task focusing on multilingual and cross-lingual sentence similarity, introduces the STS Benchmark dataset, and analyzes current model performance and limitations.

Contribution

It introduces a new multilingual and cross-lingual semantic similarity evaluation framework and the STS Benchmark dataset for improved model training and assessment.

Findings

High participation from 31 teams across language tracks

Analysis reveals common errors and limitations in current models

Introduction of the STS Benchmark dataset for future research

Abstract

Semantic Textual Similarity (STS) measures the meaning similarity of sentences. Applications include machine translation (MT), summarization, generation, question answering (QA), short answer grading, semantic search, dialog and conversational systems. The STS shared task is a venue for assessing the current state-of-the-art. The 2017 task focuses on multilingual and cross-lingual pairs with one sub-track exploring MT quality estimation (MTQE) data. The task obtained strong participation from 31 teams, with 17 participating in all language tracks. We summarize performance and review a selection of well performing methods. Analysis highlights common errors, providing insight into the limitations of existing models. To support ongoing work on semantic representations, the STS Benchmark is introduced as a new shared training and evaluation set carefully selected from the corpus of English STS shared task data (2012-2017).