Aligning Language Models from User Interactions

TL;DR

This paper introduces a scalable self-distillation method that leverages multi-turn user interactions to improve language model alignment, personalization, and continual adaptation without sacrificing other capabilities.

Contribution

It presents a novel approach to learn from user interactions directly through self-distillation, enhancing model alignment and personalization in a scalable manner.

Findings

Training on real-world user conversations improves alignment and instruction-following.

The method enables continual personalization without explicit feedback.

Models retain their general capabilities after fine-tuning with user interactions.

Abstract

Multi-turn user interactions are among the most abundant data produced by language models, yet we lack effective methods to learn from them. While typically discarded, these interactions often contain useful information: follow-up user messages may indicate that a response was incorrect, failed to follow an instruction, or did not align with the user's preferences. Importantly, language models are already able to make use of this information in context. After observing a user's follow-up, the same model is often able to revise its behavior. We leverage this ability to propose a principled and scalable method for learning directly from user interactions through self-distillation. By conditioning the model on the user's follow-up message and comparing the resulting token distribution with the original policy, we obtain a target for updating the policy that captures how the model's behavior changes in hindsight. We then distill this hindsight distribution back into the current policy. Remarkably, we show that training on real-world user conversations from WildChat improves language models across standard alignment and instruction-following benchmarks, without regressing other capabilities. The same mechanism enables personalization, allowing models to continually adapt to individual users through interaction without explicit feedback. Our results demonstrate that raw user interactions that arise naturally during deployment enable alignment, personalization, and continual adaptation.