Evaluating Style-Personalized Text Generation: Challenges and Directions

TL;DR

This paper critically examines the effectiveness of current evaluation metrics for style-personalized text generation, proposing a new benchmark and demonstrating that ensemble metrics outperform single metrics in assessing personalized writing styles.

Contribution

It introduces a comprehensive style discrimination benchmark and evaluates the limitations of existing metrics, providing guidance for more reliable assessment methods.

Findings

Ensemble evaluation metrics outperform single-metric approaches.

Current metrics show poor correlation with human judgments.

The proposed benchmark spans multiple writing tasks and evaluation settings.

Abstract

With the surge of large language models (LLMs) and their ability to produce customized output, style-personalized text generation--"write like me"--has become a rapidly growing area of interest. However, style personalization is highly specific, relative to every user, and depends strongly on the pragmatic context, which makes it uniquely challenging. Although prior research has introduced benchmarks and metrics for this area, they tend to be non-standardized and have known limitations (e.g., poor correlation with human subjects). LLMs have been found to not capture author-specific style well, it follows that the metrics themselves must be scrutinized carefully. In this work we critically examine the effectiveness of the most common metrics used in the field, such as BLEU, embeddings, and LLMs-as-judges. We evaluate these metrics using our proposed style discrimination benchmark, which spans eight diverse writing tasks across three evaluation settings: domain discrimination, authorship attribution, and LLM-generated personalized vs non-personalized discrimination. We find strong evidence that employing ensembles of diverse evaluation metrics consistently outperforms single-evaluator methods, and conclude by providing guidance on how to reliably assess style-personalized text generation.