Aligning Large Language Models with Searcher Preferences

TL;DR

This paper introduces SearchLLM, a large language model designed for open-ended generative search, utilizing a hierarchical reward system and novel training strategies to improve answer quality, safety, and user alignment.

Contribution

The work presents the first LLM tailored for open-ended generative search, with a multi-dimensional reward system and a new training method called Gated Aggregation Strategy.

Findings

Improved generation quality and user engagement in online tests.

Increased Valid Consumption Rate by 1.03%.

Reduced Re-search Rate by 2.81%.

Abstract

The paradigm shift from item-centric ranking to answer-centric synthesis is redefining the role of search engines. While recent industrial progress has applied generative techniques to closed-set item ranking in e-commerce, research and deployment of open-ended generative search on large content platforms remain limited. This setting introduces challenges, including robustness to noisy retrieval, non-negotiable safety guarantees, and alignment with diverse user needs. In this work, we introduce SearchLLM, the first large language model (LLM) for open-ended generative search. We design a hierarchical, multi-dimensional reward system that separates bottom-line constraints, including factual grounding, basic answer quality and format compliance, from behavior optimization objectives that promote robustness to noisy retrieval and alignment with user needs. Concretely, our reward model evaluates responses conditioned on the user query, session history, and retrieved evidence set, combining rule-based checks with human-calibrated LLM judges to produce an interpretable score vector over these dimensions. We introduce a Gated Aggregation Strategy to derive the training reward for optimizing SearchLLM with Group Relative Policy Optimization (GRPO). We deploy SearchLLM in the AI search entry of RedNote. Offline evaluations and online A/B tests show improved generation quality and user engagement, increasing Valid Consumption Rate by 1.03% and reducing Re-search Rate by 2.81%, while upholding strict safety and reliability standards.