AR-Med: Automated Relevance Enhancement in Medical Search via LLM-Driven Information Augmentation

TL;DR

AR-Med is a scalable framework that enhances medical search relevance by grounding LLM reasoning in verified knowledge, improving accuracy and user satisfaction in healthcare platforms.

Contribution

The paper introduces AR-Med, a retrieval-augmented LLM framework with knowledge distillation and a new benchmark, enabling reliable and efficient medical search at scale.

Findings

Achieves over 93% offline accuracy.

Improves online relevance by 24%.

Enhances user satisfaction in healthcare search.

Abstract

Accurate and reliable search on online healthcare platforms is critical for user safety and service efficacy. Traditional methods, however, often fail to comprehend complex and nuanced user queries, limiting their effectiveness. Large language models (LLMs) present a promising solution, offering powerful semantic understanding to bridge this gap. Despite their potential, deploying LLMs in this high-stakes domain is fraught with challenges, including factual hallucinations, specialized knowledge gaps, and high operational costs. To overcome these barriers, we introduce \textbf{AR-Med}, a novel framework for \textbf{A}utomated \textbf{R}elevance assessment for \textbf{Med}ical search that has been successfully deployed at scale on the Online Medical Delivery Platforms. AR-Med grounds LLM reasoning in verified medical knowledge through a retrieval-augmented approach, ensuring high accuracy and reliability. To enable efficient online service, we design a practical knowledge distillation scheme that compresses large teacher models into compact yet powerful student models. We also introduce LocalQSMed, a multi-expert annotated benchmark developed to guide model iteration and ensure strong alignment between offline and online performance. Extensive experiments show AR-Med achieves an offline accuracy of over 93\%, a 24\% absolute improvement over the original online system, and delivers significant gains in online relevance and user satisfaction. Our work presents a practical and scalable blueprint for developing trustworthy, LLM-powered systems in real-world healthcare applications.