Naturally Occurring Feedback is Common, Extractable and Useful

TL;DR

This paper demonstrates that naturally occurring user feedback in conversations is common, extractable, and beneficial for training more aligned language models, reducing the need for costly manual feedback collection.

Contribution

It introduces a method to automatically extract natural feedback from conversations, showing its effectiveness in improving model alignment and reducing feedback collection costs.

Findings

Up to 30% of chats contain explicit feedback.

Automatically extracted feedback improves model alignment.

Feedback extraction from 1M conversations yields valuable training data.

Abstract

Human feedback data is a critical component in developing language models. However, collecting this feedback is costly and ultimately not scalable. Inspired by the way human interlocutors provide spontaneous unsolicited feedback to each other, we propose to extract feedback that users naturally include when interacting with chat models. We manually annotated conversations to confirm the presence of naturally occurring feedback in a standard corpus, finding that as much as 30% of the chats include explicit feedback. Comparing to older datasets, we find that naturally occurring feedback is more prevalent in recent conversation datasets, suggesting that more than ever, naturally occurring feedback can serve as a valuable resource for feedback data. We propose a method for automatically extracting this feedback, and apply it to over 1M conversations to obtain hundreds of thousands of feedback samples. The extracted feedback shows promise: training with it improves over baseline models and enhances model alignment to human preferences.