Scaling Search Relevance: Augmenting App Store Ranking with LLM-Generated Judgments

TL;DR

This paper enhances app store search relevance by generating large-scale textual relevance labels using fine-tuned LLMs, improving ranking performance and user conversion rates, especially for tail queries.

Contribution

It introduces a method to generate high-quality textual relevance labels with LLMs, addressing data scarcity and improving search ranking effectiveness.

Findings

Offline NDCG improved for both behavioral and textual relevance

A/B testing showed a +0.24% increase in conversion rate

Significant gains in tail query performance

Abstract

Large-scale commercial search systems optimize for relevance to drive successful sessions that help users find what they are looking for. To maximize relevance, we leverage two complementary objectives: behavioral relevance (results users tend to click or download) and textual relevance (a result's semantic fit to the query). A persistent challenge is the scarcity of expert-provided textual relevance labels relative to abundant behavioral relevance labels. We first address this by systematically evaluating LLM configurations, finding that a specialized, fine-tuned model significantly outperforms a much larger pre-trained one in providing highly relevant labels. Using this optimal model as a force multiplier, we generate millions of textual relevance labels to overcome the data scarcity. We show that augmenting our production ranker with these textual relevance labels leads to a significant outward shift of the Pareto frontier: offline NDCG improves for behavioral relevance while simultaneously increasing for textual relevance. These offline gains were validated by a worldwide A/B test on the App Store ranker, which demonstrated a statistically significant +0.24% increase in conversion rate, with the most substantial performance gains occurring in tail queries, where the new textual relevance labels provide a robust signal in the absence of reliable behavioral relevance labels.