Crafting Tomorrow's Headlines: Neural News Generation and Detection in English, Turkish, Hungarian, and Persian

TL;DR

This paper introduces a multilingual benchmark dataset and evaluates various detection methods to identify machine-generated news across four languages, addressing misinformation challenges posed by advanced LLMs.

Contribution

It provides a novel benchmark dataset for neural news detection in four languages and evaluates multiple detection approaches, including linguistic features and Transformer models.

Findings

Detection performance varies across languages and models.

Transformer-based detectors show promising robustness.

Multilingual detection remains a challenging task.

Abstract

In the era dominated by information overload and its facilitation with Large Language Models (LLMs), the prevalence of misinformation poses a significant threat to public discourse and societal well-being. A critical concern at present involves the identification of machine-generated news. In this work, we take a significant step by introducing a benchmark dataset designed for neural news detection in four languages: English, Turkish, Hungarian, and Persian. The dataset incorporates outputs from multiple multilingual generators (in both, zero-shot and fine-tuned setups) such as BloomZ, LLaMa-2, Mistral, Mixtral, and GPT-4. Next, we experiment with a variety of classifiers, ranging from those based on linguistic features to advanced Transformer-based models and LLMs prompting. We present the detection results aiming to delve into the interpretablity and robustness of machine-generated texts detectors across all target languages.