Never guess what I heard... Rumor Detection in Finnish News: a Dataset and a Baseline

TL;DR

This paper introduces a Finnish rumor detection dataset and evaluates various models, finding that fine-tuned FinBERT achieves high accuracy, but performance varies due to training data differences and language-specific challenges.

Contribution

The study provides a new Finnish rumor detection dataset and benchmarks multiple models, highlighting the impact of training data and language-specific issues on model performance.

Findings

FinBERT achieves 94.3% overall accuracy.

Multilingual BERT reaches 97.2% factual label accuracy.

LSTM outperforms models trained with pretrained word2vec.

Abstract

This study presents a new dataset on rumor detection in Finnish language news headlines. We have evaluated two different LSTM based models and two different BERT models, and have found very significant differences in the results. A fine-tuned FinBERT reaches the best overall accuracy of 94.3% and rumor label accuracy of 96.0% of the time. However, a model fine-tuned on Multilingual BERT reaches the best factual label accuracy of 97.2%. Our results suggest that the performance difference is due to a difference in the original training data. Furthermore, we find that a regular LSTM model works better than one trained with a pretrained word2vec model. These findings suggest that more work needs to be done for pretrained models in Finnish language as they have been trained on small and biased corpora.