Evaluation of Multilingual LLMs Personalized Text Generation Capabilities Targeting Groups and Social-Media Platforms

TL;DR

This study evaluates how multilingual large language models personalize text generation across different languages, groups, and social media platforms, revealing variations in personalization quality and detectability, with implications for misuse and benefits.

Contribution

It extends analysis of personalization effects in multilingual LLMs across 10 languages, highlighting differences in quality and detectability for various demographic and platform targets.

Findings

Personalization quality varies across languages and targets.

Targeting social-media platforms increases detectability, especially in English.

Multilingual models show diverse personalization capabilities and risks.

Abstract

Capabilities of large language models to generate multilingual coherent text have continuously enhanced in recent years, which opens concerns about their potential misuse. Previous research has shown that they can be misused for generation of personalized disinformation in multiple languages. It has also been observed that personalization negatively affects detectability of machine-generated texts; however, this has been studied in the English language only. In this work, we examine this phenomenon across 10 languages, while we focus not only on potential misuse of personalization capabilities, but also on potential benefits they offer. Overall, we cover 1080 combinations of various personalization aspects in the prompts, for which the texts are generated by 16 distinct language models (17,280 texts in total). Our results indicate that there are differences in personalization quality of the generated texts when targeting demographic groups and when targeting social-media platforms across languages. Personalization towards platforms affects detectability of the generated texts in a higher scale, especially in English, where the personalization quality is the highest.