Faithfulness in Natural Language Generation: A Systematic Survey of Analysis, Evaluation and Optimization Methods

TL;DR

This paper systematically surveys the progress in analyzing, evaluating, and optimizing faithfulness in natural language generation, addressing the challenge of factual accuracy in generated texts across various tasks.

Contribution

It provides a comprehensive taxonomy and comparison of methods for improving faithfulness in NLG, unifying diverse approaches across tasks.

Findings

Organized evaluation and optimization methods into a unified taxonomy

Identified key research trends in faithfulness improvement

Highlighted challenges and future directions in NLG faithfulness

Abstract

Natural Language Generation (NLG) has made great progress in recent years due to the development of deep learning techniques such as pre-trained language models. This advancement has resulted in more fluent, coherent and even properties controllable (e.g. stylistic, sentiment, length etc.) generation, naturally leading to development in downstream tasks such as abstractive summarization, dialogue generation, machine translation, and data-to-text generation. However, the faithfulness problem that the generated text usually contains unfaithful or non-factual information has become the biggest challenge, which makes the performance of text generation unsatisfactory for practical applications in many real-world scenarios. Many studies on analysis, evaluation, and optimization methods for faithfulness problems have been proposed for various tasks, but have not been organized, compared and discussed in a combined manner. In this survey, we provide a systematic overview of the research progress on the faithfulness problem of NLG, including problem analysis, evaluation metrics and optimization methods. We organize the evaluation and optimization methods for different tasks into a unified taxonomy to facilitate comparison and learning across tasks. Several research trends are discussed further.