MultiSocial: Multilingual Benchmark of Machine-Generated Text Detection of Social-Media Texts

TL;DR

This paper introduces MultiSocial, a comprehensive multilingual benchmark dataset for evaluating machine-generated text detection on social-media texts across 22 languages and 5 platforms, addressing a significant gap in current research.

Contribution

It provides the first large-scale multilingual and multi-platform dataset for social-media text detection, enabling evaluation of existing methods in zero-shot and fine-tuned settings.

Findings

Fine-tuned detectors perform well on social-media texts.

Platform selection influences detection performance.

Existing detection methods can be effectively trained on social-media data.

Abstract

Recent LLMs are able to generate high-quality multilingual texts, indistinguishable for humans from authentic human-written ones. Research in machine-generated text detection is however mostly focused on the English language and longer texts, such as news articles, scientific papers or student essays. Social-media texts are usually much shorter and often feature informal language, grammatical errors, or distinct linguistic items (e.g., emoticons, hashtags). There is a gap in studying the ability of existing methods in detection of such texts, reflected also in the lack of existing multilingual benchmark datasets. To fill this gap we propose the first multilingual (22 languages) and multi-platform (5 social media platforms) dataset for benchmarking machine-generated text detection in the social-media domain, called MultiSocial. It contains 472,097 texts, of which about 58k are human-written and approximately the same amount is generated by each of 7 multilingual LLMs. We use this benchmark to compare existing detection methods in zero-shot as well as fine-tuned form. Our results indicate that the fine-tuned detectors have no problem to be trained on social-media texts and that the platform selection for training matters.