Leveraging Implicit Feedback from Deployment Data in Dialogue

TL;DR

This paper explores using implicit signals from real user interactions, such as response length and sentiment, to improve social dialogue agents without additional annotations, based on deployment data from BlenderBot.

Contribution

It introduces a method to leverage implicit feedback signals from deployment data to enhance dialogue models, highlighting the effects of different proxy signals on response quality.

Findings

Optimizing for positive sentiment reduces undesirable behaviors.

Longer responses correlate with more controversial content.

Some proxy signals can lead to undesirable response properties.

Abstract

We study improving social conversational agents by learning from natural dialogue between users and a deployed model, without extra annotations. To implicitly measure the quality of a machine-generated utterance, we leverage signals like user response length, sentiment and reaction of the future human utterances in the collected dialogue episodes. Our experiments use the publicly released deployment data from BlenderBot (Xu et al., 2023). Human evaluation indicates improvements in our new models over baseline responses; however, we find that some proxy signals can lead to more generations with undesirable properties as well. For example, optimizing for conversation length can lead to more controversial or unfriendly generations compared to the baseline, whereas optimizing for positive sentiment or reaction can decrease these behaviors.