From Understanding to Generation: An Efficient Shortcut for Evaluating Language Models

TL;DR

This paper introduces a method to convert costly language model generation tasks into cheaper understanding tasks, significantly reducing evaluation time while maintaining reliable capability assessment during training.

Contribution

The authors propose a novel reformulation of generative tasks into computationally cheaper understanding tasks, enabling faster evaluation of language models without sacrificing accuracy.

Findings

Strong correlation between original and reformulated task performance.

Over 35x average reduction in evaluation time.

Effective assessment of reasoning, coding, and knowledge capabilities.

Abstract

Iterative evaluation of LLMs during training is essential to ensure expected capability development, but can be time- and compute-intensive. While NLU tasks, where the model selects from fixed answer choices, are cheap to evaluate, essential capabilities like reasoning and code generation rely on the more time-consuming NLG (token-by-token generation) format. In this work, our aim is to decrease the computational burden of NLG benchmarks in order to enable monitoring crucial LLM capabilities during model training. We reformulate generative tasks into computationally cheaper NLU alternatives. We test the performance correlation between the original and reformulated tasks using 8 LMs of various sizes and 4 capabilities: mathematical reasoning, code generation, factual knowledge and reading comprehension. Our results show a strong correlation between task formats, supporting capability assessment via cheaper alternatives and achieving over 35x average reduction in evaluation time. Our project is available at: https://github.com/Fraunhofer-IIS/EvalShortcut