Semantic Search At LinkedIn

TL;DR

LinkedIn developed an efficient LLM-based semantic search system that combines relevance judgment, embedding retrieval, and a compact model to improve search quality and engagement while maintaining high inference efficiency.

Contribution

The paper introduces a novel semantic search framework integrating a relevance judge, embedding retrieval, and a small distilled language model, achieving high efficiency and quality in production.

Findings

Over 75x increase in ranking throughput with maintained NDCG

Successful deployment of LLM-based ranking in production

Significant improvements in user engagement and search relevance

Abstract

Semantic search with large language models (LLMs) enables retrieval by meaning rather than keyword overlap, but scaling it requires major inference efficiency advances. We present LinkedIn's LLM-based semantic search framework for AI Job Search and AI People Search, combining an LLM relevance judge, embedding-based retrieval, and a compact Small Language Model trained via multi-teacher distillation to jointly optimize relevance and engagement. A prefill-oriented inference architecture co-designed with model pruning, context compression, and text-embedding hybrid interactions boosts ranking throughput by over 75x under a fixed latency constraint while preserving near-teacher-level NDCG, enabling one of the first production LLM-based ranking systems with efficiency comparable to traditional approaches and delivering significant gains in quality and user engagement.