FLAIR: Feedback Learning for Adaptive Information Retrieval

TL;DR

FLAIR is a feedback-driven framework that enhances large language model copilots' information retrieval by integrating expert feedback, leading to improved accuracy and scalability in real-world applications.

Contribution

Introduces FLAIR, a novel feedback learning framework that adapts retrieval strategies using domain-specific expert feedback for large language model copilots.

Findings

Significant performance improvements over state-of-the-art methods

Effective in both seen and unseen query scenarios

Successfully deployed at Microsoft with thousands of users

Abstract

Recent advances in Large Language Models (LLMs) have driven the adoption of copilots in complex technical scenarios, underscoring the growing need for specialized information retrieval solutions. In this paper, we introduce FLAIR, a lightweight, feedback learning framework that adapts copilot systems' retrieval strategies by integrating domain-specific expert feedback. FLAIR operates in two stages: an offline phase obtains indicators from (1) user feedback and (2) questions synthesized from documentation, storing these indicators in a decentralized manner. An online phase then employs a two-track ranking mechanism to combine raw similarity scores with the collected indicators. This iterative setup refines retrieval performance for any query. Extensive real-world evaluations of FLAIR demonstrate significant performance gains on both previously seen and unseen queries, surpassing state-of-the-art approaches. The system has been successfully integrated into Copilot DECO, serving thousands of users at Microsoft, demonstrating its scalability and effectiveness in operational environments.