Human Judgement as a Compass to Navigate Automatic Metrics for Formality Transfer

TL;DR

This paper evaluates automatic metrics for formality transfer by comparing them with human judgments, analyzing their correlation across style strength, content preservation, and fluency, and providing guidelines for their use.

Contribution

It offers a comprehensive analysis of how automatic metrics align with human judgments in formality transfer, and provides recommendations for their application.

Findings

Certain automatic metrics correlate well with human judgments in style strength.

Content preservation metrics show moderate correlation with human assessments.

Guidelines are proposed for selecting appropriate metrics in formality transfer evaluation.

Abstract

Although text style transfer has witnessed rapid development in recent years, there is as yet no established standard for evaluation, which is performed using several automatic metrics, lacking the possibility of always resorting to human judgement. We focus on the task of formality transfer, and on the three aspects that are usually evaluated: style strength, content preservation, and fluency. To cast light on how such aspects are assessed by common and new metrics, we run a human-based evaluation and perform a rich correlation analysis. We are then able to offer some recommendations on the use of such metrics in formality transfer, also with an eye to their generalisability (or not) to related tasks.