Naturalness Evaluation of Natural Language Generation in Task-oriented Dialogues using BERT

TL;DR

This paper introduces an automatic BERT-based method for evaluating the naturalness of language generated by dialogue systems, outperforming traditional baselines and leveraging transfer learning for improved efficiency.

Contribution

The paper proposes a novel automatic naturalness evaluation method for dialogue NLG using BERT, demonstrating superior performance and faster training through transfer learning.

Findings

BERT-based evaluation outperforms SVM, BiLSTM, and BLEURT baselines.

Transfer learning enhances training speed and evaluation accuracy.

The method provides a cost-effective alternative to human evaluation.

Abstract

This paper presents an automatic method to evaluate the naturalness of natural language generation in dialogue systems. While this task was previously rendered through expensive and time-consuming human labor, we present this novel task of automatic naturalness evaluation of generated language. By fine-tuning the BERT model, our proposed naturalness evaluation method shows robust results and outperforms the baselines: support vector machines, bi-directional LSTMs, and BLEURT. In addition, the training speed and evaluation performance of naturalness model are improved by transfer learning from quality and informativeness linguistic knowledge.