Investigating label suggestions for opinion mining in German Covid-19 social media

TL;DR

This study explores how interactively updated label suggestions can enhance opinion mining annotation efficiency in German Covid-19 social media data, showing that small expert-trained models significantly improve annotation consistency and quality.

Contribution

It demonstrates that small, expert-annotated models provide effective label suggestions that improve annotation agreement and quality in opinion mining tasks.

Findings

Expert-trained models improve annotation agreement (+.14 Fleiss' κ)

Static model suggestions are as effective as interactively trained models

Annotated data is suitable for transfer learning experiments

Abstract

This work investigates the use of interactively updated label suggestions to improve upon the efficiency of gathering annotations on the task of opinion mining in German Covid-19 social media data. We develop guidelines to conduct a controlled annotation study with social science students and find that suggestions from a model trained on a small, expert-annotated dataset already lead to a substantial improvement - in terms of inter-annotator agreement(+.14 Fleiss' $\kappa$) and annotation quality - compared to students that do not receive any label suggestions. We further find that label suggestions from interactively trained models do not lead to an improvement over suggestions from a static model. Nonetheless, our analysis of suggestion bias shows that annotators remain capable of reflecting upon the suggested label in general. Finally, we confirm the quality of the annotated data in transfer learning experiments between different annotator groups. To facilitate further research in opinion mining on social media data, we release our collected data consisting of 200 expert and 2,785 student annotations.