Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation

TL;DR

This survey reviews recent research on using human feedback to enhance natural language generation, formalizing feedback, organizing existing work, and exploring AI feedback to reduce human involvement.

Contribution

It provides a comprehensive formalization of feedback, taxonomy of research, and discusses approaches, datasets, concerns, and emerging AI feedback methods in natural language generation.

Findings

Feedback improves model quality and safety

Two main approaches: training feedback models and using feedback for decoding

Emerging AI feedback reduces human intervention

Abstract

Many recent advances in natural language generation have been fueled by training large language models on internet-scale data. However, this paradigm can lead to models that generate toxic, inaccurate, and unhelpful content, and automatic evaluation metrics often fail to identify these behaviors. As models become more capable, human feedback is an invaluable signal for evaluating and improving models. This survey aims to provide an overview of the recent research that has leveraged human feedback to improve natural language generation. First, we introduce an encompassing formalization of feedback, and identify and organize existing research into a taxonomy following this formalization. Next, we discuss how feedback can be described by its format and objective, and cover the two approaches proposed to use feedback (either for training or decoding): directly using the feedback or training feedback models. We also discuss existing datasets for human-feedback data collection, and concerns surrounding feedback collection. Finally, we provide an overview of the nascent field of AI feedback, which exploits large language models to make judgments based on a set of principles and minimize the need for human intervention.