Multidimensional Evaluation for Text Style Transfer Using ChatGPT

TL;DR

This paper explores ChatGPT's effectiveness as a multidimensional evaluator for Text Style Transfer, comparing it to existing metrics and human judgments in a zero-shot setting across style strength, content preservation, and fluency.

Contribution

It demonstrates that ChatGPT can serve as a competitive automatic evaluator for multiple dimensions of text style transfer, offering insights into large language models' evaluation capabilities.

Findings

ChatGPT shows competitive correlation with human judgments.

It performs well across style strength, content preservation, and fluency.

Preliminary results suggest potential for LLMs in multidimensional evaluation.

Abstract

We investigate the potential of ChatGPT as a multidimensional evaluator for the task of \emph{Text Style Transfer}, alongside, and in comparison to, existing automatic metrics as well as human judgements. We focus on a zero-shot setting, i.e. prompting ChatGPT with specific task instructions, and test its performance on three commonly-used dimensions of text style transfer evaluation: style strength, content preservation, and fluency. We perform a comprehensive correlation analysis for two transfer directions (and overall) at different levels. Compared to existing automatic metrics, ChatGPT achieves competitive correlations with human judgments. These preliminary results are expected to provide a first glimpse into the role of large language models in the multidimensional evaluation of stylized text generation.