User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signal

TL;DR

This paper investigates how implicit user feedback from long-term interactions with language models can serve as a learning signal, revealing its potential and limitations through analysis of real interaction datasets.

Contribution

It provides an in-depth analysis of implicit user feedback in human-LLM dialogues and explores its effectiveness as a learning signal for model improvement.

Findings

Feedback helps improve performance on short questions

Feedback is less effective on complex, longer questions

Implicit feedback has potential but also notable limitations

Abstract

Once language models (LMs) are deployed, they can interact with users long-term, ideally evolving based on their feedback. Asking for direct user feedback can be disruptive; thus, we study harvesting implicit user feedback from user-LM interaction logs. We study two user-LM interaction datasets (WildChat and LMSYS). First, we analyze user feedback in the user-LLM conversation logs, providing insights into when and why such feedback occurs. Second, we study harvesting learning signals from such implicit user feedback. Specifically, we study whether incorporating the contents of user feedback (e.g., user wanted clarification), in addition to the polarity of the feedback, can improve the model performance. We observe mixed results, showing this helps in short human-designed questions (MTBench) but not on longer and more complex questions (WildBench). Together, we provide an in-depth study of implicit user feedback, showing its potential and limitations.