DMA: Online RAG Alignment with Human Feedback

TL;DR

DMA introduces an online learning framework for retrieval-augmented generation systems that leverages multi-granularity human feedback to adapt in real-time, improving engagement and performance without sacrificing baseline capabilities.

Contribution

The paper presents DMA, a novel online learning framework that systematically incorporates human feedback at multiple levels to improve RAG systems' adaptability and performance.

Findings

DMA improves human engagement in industrial deployment.

DMA maintains competitive retrieval performance while enhancing conversational QA.

Online and offline evaluations demonstrate DMA's effectiveness in real-time adaptation.

Abstract

Retrieval-augmented generation (RAG) systems often rely on static retrieval, limiting adaptation to evolving intent and content drift. We introduce Dynamic Memory Alignment (DMA), an online learning framework that systematically incorporates multi-granularity human feedback to align ranking in interactive settings. DMA organizes document-, list-, and response-level signals into a coherent learning pipeline: supervised training for pointwise and listwise rankers, policy optimization driven by response-level preferences, and knowledge distillation into a lightweight scorer for low-latency serving. Throughout this paper, memory refers to the model's working memory, which is the entire context visible to the LLM for In-Context Learning. We adopt a dual-track evaluation protocol mirroring deployment: (i) large-scale online A/B ablations to isolate the utility of each feedback source, and (ii) few-shot offline tests on knowledge-intensive benchmarks. Online, a multi-month industrial deployment further shows substantial improvements in human engagement. Offline, DMA preserves competitive foundational retrieval while yielding notable gains on conversational QA (TriviaQA, HotpotQA). Taken together, these results position DMA as a principled approach to feedback-driven, real-time adaptation in RAG without sacrificing baseline capability.