Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Study

TL;DR

This study investigates the effectiveness of large language models, especially ChatGPT, in reference-free text quality evaluation, demonstrating their superior performance over existing automatic metrics and proposing an explicit scoring method.

Contribution

The paper introduces a novel evaluation approach using ChatGPT for reference-free text quality assessment and compares three methods, highlighting the effectiveness of the Explicit Score.

Findings

ChatGPT outperforms most automatic metrics in text quality evaluation.

The Explicit Score method provides the most reliable and effective assessment.

Direct comparison of two texts using ChatGPT may sometimes be suboptimal.

Abstract

Evaluating the quality of generated text is a challenging task in NLP, due to the inherent complexity and diversity of text. Recently, large language models (LLMs) have garnered significant attention due to their impressive performance in various tasks. Therefore, we present this paper to investigate the effectiveness of LLMs, especially ChatGPT, and explore ways to optimize their use in assessing text quality. We compared three kinds of reference-free evaluation methods. The experimental results prove that ChatGPT is capable of evaluating text quality effectively from various perspectives without reference and demonstrates superior performance than most existing automatic metrics. In particular, the Explicit Score, which utilizes ChatGPT to generate a numeric score measuring text quality, is the most effective and reliable method among the three exploited approaches. However, directly comparing the quality of two texts may lead to suboptimal results. We believe this paper will provide valuable insights for evaluating text quality with LLMs and have released the used data.