Improve Large Language Model Systems with User Logs

TL;DR

This paper introduces UNO, a framework that leverages user logs to enhance large language models by distilling feedback, managing data heterogeneity, and filtering noise, leading to improved performance.

Contribution

The paper presents UNO, a novel unified approach for improving LLM systems using user logs, addressing noise and heterogeneity challenges in real-world data.

Findings

UNO outperforms RAG and memory-based baselines in effectiveness.

The framework achieves state-of-the-art efficiency in LLM improvement.

Extensive experiments validate UNO's ability to filter noise and adapt to user feedback.

Abstract

Scaling training data and model parameters has long driven progress in large language models (LLMs), but this paradigm is increasingly constrained by the scarcity of high-quality data and diminishing returns from rising computational costs. As a result, recent work is increasing the focus on continual learning from real-world deployment, where user interaction logs provide a rich source of authentic human feedback and procedural knowledge. However, learning from user logs is challenging due to their unstructured and noisy nature. Vanilla LLM systems often struggle to distinguish useful feedback signals from noisy user behavior, and the disparity between user log collection and model optimization (e.g., the off-policy optimization problem) further strengthens the problem. To this end, we propose UNO (User log-driveN Optimization), a unified framework for improving LLM systems (LLMsys) with user logs. UNO first distills logs into semi-structured rules and preference pairs, then employs query-and-feedback-driven clustering to manage data heterogeneity, and finally quantifies the cognitive gap between the model's prior knowledge and the log data. This assessment guides the LLMsys to adaptively filter out noisy feedback and construct different modules for primary and reflective experiences extracted from user logs, thereby improving future responses. Extensive experiments show that UNO achieves state-of-the-art effectiveness and efficiency, significantly outperforming Retrieval Augmented Generation (RAG) and memory-based baselines. We have open-sourced our code at https://github.com/bebr2/UNO .