SaRoCo: Detecting Satire in a Novel Romanian Corpus of News Articles

TL;DR

This paper introduces a large Romanian satire news corpus and evaluates deep neural models, revealing significant room for improvement in machine satire detection compared to human performance.

Contribution

It provides the first large-scale Romanian satire news corpus with a proper train-test split and baseline results using state-of-the-art neural models.

Findings

Machine accuracy under 73% on satire detection

Human accuracy around 87%

Baseline models establish strong benchmarks for future research

Abstract

In this work, we introduce a corpus for satire detection in Romanian news. We gathered 55,608 public news articles from multiple real and satirical news sources, composing one of the largest corpora for satire detection regardless of language and the only one for the Romanian language. We provide an official split of the text samples, such that training news articles belong to different sources than test news articles, thus ensuring that models do not achieve high performance simply due to overfitting. We conduct experiments with two state-of-the-art deep neural models, resulting in a set of strong baselines for our novel corpus. Our results show that the machine-level accuracy for satire detection in Romanian is quite low (under 73% on the test set) compared to the human-level accuracy (87%), leaving enough room for improvement in future research.