Uralic Language Identification (ULI) 2020 shared task dataset and the Wanca 2017 corpus

TL;DR

This paper presents the Wanca 2017 corpus and the Uralic Language Identification (ULI) 2020 dataset, enabling improved language identification for rare Uralic languages through new data and baseline experiments.

Contribution

It introduces the Wanca 2017 corpus and details the construction of the ULI 2020 dataset for rare Uralic languages, along with baseline identification results.

Findings

Baseline experiments demonstrate the dataset's utility for language identification.

The dataset includes texts from Wanca 2017 and Leipzig corpora collection.

Provides a resource for future research on Uralic language identification.

Abstract

This article introduces the Wanca 2017 corpus of texts crawled from the internet from which the sentences in rare Uralic languages for the use of the Uralic Language Identification (ULI) 2020 shared task were collected. We describe the ULI dataset and how it was constructed using the Wanca 2017 corpus and texts in different languages from the Leipzig corpora collection. We also provide baseline language identification experiments conducted using the ULI 2020 dataset.