AutoEval Done Right: Using Synthetic Data for Model Evaluation

TL;DR

This paper introduces statistically principled algorithms for autoevaluation using synthetic data, significantly reducing the need for human annotations and increasing sample efficiency in model evaluation.

Contribution

It presents novel algorithms that improve sample efficiency and remain unbiased for autoevaluation with synthetic data, demonstrated on GPT-4 experiments.

Findings

Sample efficiency increased by up to 50% with synthetic data.

Algorithms remain unbiased while improving evaluation efficiency.

Effective reduction in human annotation requirements.

Abstract

The evaluation of machine learning models using human-labeled validation data can be expensive and time-consuming. AI-labeled synthetic data can be used to decrease the number of human annotations required for this purpose in a process called autoevaluation. We suggest efficient and statistically principled algorithms for this purpose that improve sample efficiency while remaining unbiased. These algorithms increase the effective human-labeled sample size by up to 50% on experiments with GPT-4.