Agent4Ranking: Semantic Robust Ranking via Personalized Query Rewriting Using Multi-agent LLM

TL;DR

This paper introduces a demographic-aware query rewriting framework using multi-agent LLMs and a robust ranking architecture to improve search engine robustness against diverse user query formulations.

Contribution

It proposes a novel multi-agent LLM-based query rewriting pipeline combined with a robust MMoE ranking model, addressing demographic diversity in user queries.

Findings

Enhanced ranking accuracy across diverse demographic query formulations

Improved robustness of search results against query variations

Effective query rewriting using LLM agents emulating different demographics

Abstract

Search engines are crucial as they provide an efficient and easy way to access vast amounts of information on the internet for diverse information needs. User queries, even with a specific need, can differ significantly. Prior research has explored the resilience of ranking models against typical query variations like paraphrasing, misspellings, and order changes. Yet, these works overlook how diverse demographics uniquely formulate identical queries. For instance, older individuals tend to construct queries more naturally and in varied order compared to other groups. This demographic diversity necessitates enhancing the adaptability of ranking models to diverse query formulations. To this end, in this paper, we propose a framework that integrates a novel rewriting pipeline that rewrites queries from various demographic perspectives and a novel framework to enhance ranking robustness. To be specific, we use Chain of Thought (CoT) technology to utilize Large Language Models (LLMs) as agents to emulate various demographic profiles, then use them for efficient query rewriting, and we innovate a robust Multi-gate Mixture of Experts (MMoE) architecture coupled with a hybrid loss function, collectively strengthening the ranking models' robustness. Our extensive experimentation on both public and industrial datasets assesses the efficacy of our query rewriting approach and the enhanced accuracy and robustness of the ranking model. The findings highlight the sophistication and effectiveness of our proposed model.