RUSSE'2018: A Shared Task on Word Sense Induction for the Russian Language

TL;DR

This paper reports on the first shared task for Russian word sense induction, evaluating various models' ability to cluster contexts of ambiguous words, with 18 teams participating and outperforming previous baselines.

Contribution

It introduces new datasets and benchmarks for Russian WSI, and demonstrates the effectiveness of sense embedding models in a Slavic language context.

Findings

Multiple models outperformed previous state-of-the-art baselines.

New datasets based on Wikipedia, academic texts, and dictionaries were created.

18 teams participated, submitting 383 models.

Abstract

The paper describes the results of the first shared task on word sense induction (WSI) for the Russian language. While similar shared tasks were conducted in the past for some Romance and Germanic languages, we explore the performance of sense induction and disambiguation methods for a Slavic language that shares many features with other Slavic languages, such as rich morphology and virtually free word order. The participants were asked to group contexts of a given word in accordance with its senses that were not provided beforehand. For instance, given a word "bank" and a set of contexts for this word, e.g. "bank is a financial institution that accepts deposits" and "river bank is a slope beside a body of water", a participant was asked to cluster such contexts in the unknown in advance number of clusters corresponding to, in this case, the "company" and the "area" senses of the word "bank". For the purpose of this evaluation campaign, we developed three new evaluation datasets based on sense inventories that have different sense granularity. The contexts in these datasets were sampled from texts of Wikipedia, the academic corpus of Russian, and an explanatory dictionary of Russian. Overall, 18 teams participated in the competition submitting 383 models. Multiple teams managed to substantially outperform competitive state-of-the-art baselines from the previous years based on sense embeddings.