Persona-Augmented Benchmarking: Evaluating LLMs Across Diverse Writing Styles

TL;DR

This paper introduces a persona-based prompting method to evaluate LLMs across diverse writing styles, revealing that style variations significantly affect performance and highlighting the need for style-diverse benchmarks.

Contribution

It proposes a scalable persona-augmented benchmarking approach that enhances evaluation diversity and reveals style-dependent performance variations in LLMs.

Findings

Writing style variations significantly impact LLM performance.

Distinct styles consistently trigger different performance levels.

Style diversity in benchmarks improves external validity.

Abstract

Current benchmarks for evaluating Large Language Models (LLMs) often do not exhibit enough writing style diversity, with many adhering primarily to standardized conventions. Such benchmarks do not fully capture the rich variety of communication patterns exhibited by humans. Thus, it is possible that LLMs, which are optimized on these benchmarks, may demonstrate brittle performance when faced with "non-standard" input. In this work, we test this hypothesis by rewriting evaluation prompts using persona-based LLM prompting, a low-cost method to emulate diverse writing styles. Our results show that, even with identical semantic content, variations in writing style and prompt formatting significantly impact the estimated performance of the LLM under evaluation. Notably, we identify distinct writing styles that consistently trigger either low or high performance across a range of models and tasks, irrespective of model family, size, and recency. Our work offers a scalable approach to augment existing benchmarks, improving the external validity of the assessments they provide for measuring LLM performance across linguistic variations.