Models Know Models Best: Evaluation via Model-Preferred Formats

TL;DR

This paper investigates how different evaluation formats affect Large Language Model performance and introduces a dynamic, model-preference-based approach to optimize format selection, significantly improving zero-shot accuracy.

Contribution

It presents a novel, lightweight classifier that dynamically chooses the best evaluation format based on model signals, outperforming traditional heuristics.

Findings

Format choice significantly impacts LLM performance.

Model-preference signals can guide optimal format selection.

Dynamic format alignment improves zero-shot accuracy.

Abstract

Performance of Large Language Models (LLMs) on multiple-choice tasks differs markedly between symbol-based and cloze-style evaluation formats. The observed discrepancies are systematically attributable to task characteristics: natural language continuation benefits from likelihood scoring, whereas explicit comparison is better suited to symbol-based selection. These trends are consistent across various decoder-based LLMs, indicating model-agnostic effects. To address these inconsistencies, a dynamic format-alignment strategy is introduced that employs a lightweight classifier trained on latent model-preference signals. In contrast to human-designed heuristics, which often degrade performance, this approach uses model-generated signals to determine the optimal format for each problem instance. The proposed method achieves substantial and consistent improvements in zero-shot accuracy across reasoning and knowledge benchmarks, better revealing the models' latent capabilities.