Czech News Dataset for Semantic Textual Similarity

TL;DR

This paper introduces a large Czech news sentence dataset with semantic similarity annotations, detailing its collection, annotation process, and a baseline system demonstrating high predictive accuracy.

Contribution

It presents a new Czech semantic textual similarity dataset with extensive annotations and a baseline model outperforming average annotators.

Findings

High inter-annotator agreement (0.86)

Baseline model achieves 0.92 correlation

Dataset contains 138,556 annotations from 485 students

Abstract

This paper describes a novel dataset consisting of sentences with semantic similarity annotations. The data originate from the journalistic domain in the Czech language. We describe the process of collecting and annotating the data in detail. The dataset contains 138,556 human annotations divided into train and test sets. In total, 485 journalism students participated in the creation process. To increase the reliability of the test set, we compute the annotation as an average of 9 individual annotations. We evaluate the quality of the dataset by measuring inter and intra annotation annotators' agreements. Beside agreement numbers, we provide detailed statistics of the collected dataset. We conclude our paper with a baseline experiment of building a system for predicting the semantic similarity of sentences. Due to the massive number of training annotations (116 956), the model can perform significantly better than an average annotator (0,92 versus 0,86 of Person's correlation coefficients).