Autonomous Evaluation of LLMs for Truth Maintenance and Reasoning Tasks

TL;DR

AutoEval introduces an automated, scalable benchmark for assessing LLMs on formal tasks like truth maintenance and reasoning, reducing reliance on human labels and static datasets.

Contribution

This paper presents AutoEval, the first benchmark enabling objective, automated evaluation of LLMs across increasing difficulty levels without human annotation.

Findings

AutoEval correlates well with other benchmarks.

It effectively mitigates overfitting to static datasets.

AutoEval can evaluate models of varying sophistication.

Abstract

This paper presents AutoEval, a novel benchmark for scaling Large Language Model (LLM) assessment in formal tasks with clear notions of correctness, such as truth maintenance in translation and logical reasoning. AutoEval is the first benchmarking paradigm that offers several key advantages necessary for scaling objective evaluation of LLMs without human labeling: (a) ability to evaluate LLMs of increasing sophistication by auto-generating tasks at different levels of difficulty; (b) auto-generation of ground truth that eliminates dependence on expensive and time-consuming human annotation; (c) the use of automatically generated, randomized datasets that mitigate the ability of successive LLMs to overfit to static datasets used in many contemporary benchmarks. Empirical analysis shows that an LLM's performance on AutoEval is highly indicative of its performance on a diverse array of other benchmarks focusing on translation and reasoning tasks, making it a valuable autonomous evaluation paradigm in settings where hand-curated datasets can be hard to obtain and/or update.