SAGE: Scalable AI Governance & Evaluation

TL;DR

SAGE is a scalable framework that enhances AI governance and relevance evaluation in large-scale search systems by combining human judgment, language models, and distillation techniques, leading to improved model oversight and user engagement.

Contribution

SAGE introduces a novel calibration loop integrating human policies, precedent, and language models to produce high-quality, scalable relevance judgments for AI systems.

Findings

Achieved near human-level agreement in relevance judgments.

Reduced evaluation costs by 92 times through distillation.

Enabled detection of regressions and policy oversight in production.

Abstract

Evaluating relevance in large-scale search systems is fundamentally constrained by the governance gap between nuanced, resource-constrained human oversight and the high-throughput requirements of production systems. While traditional approaches rely on engagement proxies or sparse manual review, these methods often fail to capture the full scope of high-impact relevance failures. We present \textbf{SAGE} (Scalable AI Governance \& Evaluation), a framework that operationalizes high-quality human product judgment as a scalable evaluation signal. At the core of SAGE is a bidirectional calibration loop where natural-language \emph{Policy}, curated \emph{Precedent}, and an \emph{LLM Surrogate Judge} co-evolve. SAGE systematically resolves semantic ambiguities and misalignments, transforming subjective relevance judgment into an executable, multi-dimensional rubric with near human-level agreement. To bridge the gap between frontier model reasoning and industrial-scale inference, we apply teacher-student distillation to transfer high-fidelity judgments into compact student surrogates at \textbf{92$\times$} lower cost. Deployed within LinkedIn Search ecosystems, SAGE guided model iteration through simulation-driven development, distilling policy-aligned models for online serving and enabling rapid offline evaluation. In production, it powered policy oversight that measured ramped model variants and detected regressions invisible to engagement metrics. Collectively, these drove a \textbf{0.25\%} lift in LinkedIn daily active users.