Luna-2: Scalable Single-Token Evaluation with Small Language Models

TL;DR

Luna-2 introduces a scalable, deterministic evaluation architecture using small language models with lightweight adapters, achieving high accuracy in task-specific metrics while significantly reducing cost and latency.

Contribution

The paper presents Luna-2, a novel architecture that enables fast, accurate, and cost-effective evaluation of language models using small models with specialized adapters.

Findings

Matches state-of-the-art accuracy in safety and hallucination benchmarks

Reduces inference cost by over 80 times

Decreases latency by over 20 times

Abstract

Real-time guardrails require evaluation that is accurate, cheap, and fast - yet today's default, LLM-as-a-judge (LLMAJ), is slow, expensive, and operationally non-deterministic due to multi-token generation. We present Luna-2, a novel architecture that leverages decoder-only small language models (SLMs) into a deterministic evaluation model to reliably compute complex task-specific LLMAJ metrics (e.g. toxicity, hallucination, tool selection quality, etc.) at an accuracy at par or higher than LLMAJ using frontier LLMs while drastically reducing the cost and latency of computation. Each metric is implemented as a lightweight LoRA/PEFT head on top of a shared SLM backbone, enabling hundreds of specialized metrics to run concurrently on a single GPU, deployable locally next to AI systems in a privacy-preserving and latency optimizing manner. Across content safety and hallucination benchmarks, Luna-2 matches the accuracy of state-of-the-art LLM-based evaluators while reducing inference cost by over 80x and latency by over 20x. In this paper, we outline the model architecture, training methodology and report real-world empirical results on accuracy, latency, and throughput results. In production, Luna-2 is protecting 100M+ AI sessions and processing over 100B tokens per month for our customers with eval cost savings of over $30M annually.