CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation

TL;DR

CTRLEval is an unsupervised, reference-free metric that evaluates controlled text generation by leveraging multiple text infilling tasks and pre-trained language models, showing superior correlation with human judgments and better generalization.

Contribution

It introduces a novel unsupervised, reference-free evaluation metric for controlled text generation that does not require model training and outperforms existing metrics in correlation and generalization.

Findings

Higher correlation with human judgments than baselines

Better generalization across different models and qualities

No model training required for evaluation

Abstract

Existing reference-free metrics have obvious limitations for evaluating controlled text generation models. Unsupervised metrics can only provide a task-agnostic evaluation result which correlates weakly with human judgments, whereas supervised ones may overfit task-specific data with poor generalization ability to other datasets. In this paper, we propose an unsupervised reference-free metric called CTRLEval, which evaluates controlled text generation from different aspects by formulating each aspect into multiple text infilling tasks. On top of these tasks, the metric assembles the generation probabilities from a pre-trained language model without any model training. Experimental results show that our metric has higher correlations with human judgments than other baselines, while obtaining better generalization of evaluating generated texts from different models and with different qualities.