AutoMetrics: Approximate Human Judgements with Automatically Generated Evaluators

TL;DR

AutoMetrics is a framework that synthesizes automatic evaluation metrics for AI applications by combining curated metrics and lightweight human feedback, significantly improving correlation with human judgments in diverse tasks.

Contribution

AutoMetrics introduces a novel approach to approximate human evaluations using automatically generated evaluators and a curated metric bank, reducing reliance on extensive human feedback.

Findings

Improves Kendall correlation with human ratings by up to 33.4%.

Requires fewer than 100 feedback points for effective evaluation.

Applicable across diverse AI tasks with enhanced accuracy.

Abstract

Evaluating user-facing AI applications remains a central challenge, especially in open-ended domains such as travel planning, clinical note generation, or dialogue. The gold standard is user feedback (e.g., thumbs up/down) or behavioral signals (e.g., retention), but these are often scarce in prototypes and research projects, or too-slow to use for system optimization. We present AutoMetrics, a framework for synthesizing evaluation metrics under low-data constraints. AutoMetrics combines retrieval from MetricBank, a collection of 48 metrics we curate, with automatically generated LLM-as-a-Judge criteria informed by lightweight human feedback. These metrics are composed via regression to maximize correlation with human signal. AutoMetrics takes you from expensive measures to interpretable automatic metrics. Across 5 diverse tasks, AutoMetrics improves Kendall correlation with human ratings by up to 33.4% over LLM-as-a-Judge while requiring fewer than 100 feedback points. We show that AutoMetrics can be used as a proxy reward to equal effect as a verifiable reward. We release the full AutoMetrics toolkit and MetricBank to accelerate adaptive evaluation of LLM applications.