RuArg-2022: Argument Mining Evaluation

TL;DR

This paper reports on the first Russian argumentation analysis competition, introducing a new dataset and baseline system that achieved notable F1-scores in stance detection and argument classification tasks.

Contribution

It provides the first annotated corpus and evaluation benchmarks for argument mining in Russian, utilizing a BERT-based system with translation and masking techniques.

Findings

The winning system achieved an F1-score of 0.6968 in stance detection.

The system achieved an F1-score of 0.7404 in argument classification.

The dataset and baselines will support future research in Russian argument mining.

Abstract

Argumentation analysis is a field of computational linguistics that studies methods for extracting arguments from texts and the relationships between them, as well as building argumentation structure of texts. This paper is a report of the organizers on the first competition of argumentation analysis systems dealing with Russian language texts within the framework of the Dialogue conference. During the competition, the participants were offered two tasks: stance detection and argument classification. A corpus containing 9,550 sentences (comments on social media posts) on three topics related to the COVID-19 pandemic (vaccination, quarantine, and wearing masks) was prepared, annotated, and used for training and testing. The system that won the first place in both tasks used the NLI (Natural Language Inference) variant of the BERT architecture, automatic translation into English to apply a specialized BERT model, retrained on Twitter posts discussing COVID-19, as well as additional masking of target entities. This system showed the following results: for the stance detection task an F1-score of 0.6968, for the argument classification task an F1-score of 0.7404. We hope that the prepared dataset and baselines will help to foster further research on argument mining for the Russian language.