Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer

TL;DR

This paper assesses automatic evaluation metrics for style transfer across multiple languages, identifying best practices and models that align well with human judgments to improve the development of multilingual style transfer systems.

Contribution

It provides the first multilingual evaluation of style transfer metrics, expanding beyond English to include Brazilian-Portuguese, French, and Italian, and offers guidelines for robust automatic evaluation.

Findings

Identified metrics that correlate well with human judgments across languages

Established best practices for automatic evaluation in multilingual style transfer

Highlighted the robustness of certain models across different languages

Abstract

While the field of style transfer (ST) has been growing rapidly, it has been hampered by a lack of standardized practices for automatic evaluation. In this paper, we evaluate leading ST automatic metrics on the oft-researched task of formality style transfer. Unlike previous evaluations, which focus solely on English, we expand our focus to Brazilian-Portuguese, French, and Italian, making this work the first multilingual evaluation of metrics in ST. We outline best practices for automatic evaluation in (formality) style transfer and identify several models that correlate well with human judgments and are robust across languages. We hope that this work will help accelerate development in ST, where human evaluation is often challenging to collect.