Analysing Differences in Persuasive Language in LLM-Generated Text: Uncovering Stereotypical Gender Patterns

TL;DR

This study investigates how large language models generate persuasive language, revealing significant gender-based differences that reflect stereotypical biases, through a comprehensive evaluation across multiple models and languages.

Contribution

It introduces a framework for analyzing persuasive language in LLMs considering recipient gender, sender intent, and language, highlighting gender biases in generated content.

Findings

Gender differences in persuasive language across all models

Biases align with gender-stereotypical linguistic tendencies

Framework applicable to multiple languages and models

Abstract

Large language models (LLMs) are increasingly used for everyday communication tasks, including drafting interpersonal messages intended to influence and persuade. Prior work has shown that LLMs can successfully persuade humans and amplify persuasive language. It is therefore essential to understand how user instructions affect the generation of persuasive language, and to understand whether the generated persuasive language differs, for example, when targeting different groups. In this work, we propose a framework for evaluating how persuasive language generation is affected by recipient gender, sender intent, or output language. We evaluate 13 LLMs and 16 languages using pairwise prompt instructions. We evaluate model responses on 19 categories of persuasive language using an LLM-as-judge setup grounded in social psychology and communication science. Our results reveal significant gender differences in the persuasive language generated across all models. These patterns reflect biases consistent with gender-stereotypical linguistic tendencies documented in social psychology and sociolinguistics.