"My Answer is C": First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Models

TL;DR

This paper demonstrates that first-token probability-based evaluations of instruction-tuned language models often do not accurately reflect their final generated responses, highlighting the need for output inspection.

Contribution

The study systematically compares first-token evaluation with actual text output, revealing significant misalignments and limitations of current evaluation methods for instruction-tuned models.

Findings

Over 60% mismatch between first-token and final output evaluations.

Models, especially fine-tuned ones, show high misalignment even with constrained prompts.

First-token evaluation is insufficient; output inspection is necessary.

Abstract

The open-ended nature of language generation makes the evaluation of autoregressive large language models (LLMs) challenging. One common evaluation approach uses multiple-choice questions (MCQ) to limit the response space. The model is then evaluated by ranking the candidate answers by the log probability of the first token prediction. However, first-tokens may not consistently reflect the final response output, due to model's diverse response styles such as starting with "Sure" or refusing to answer. Consequently, MCQ evaluation is not indicative of model behaviour when interacting with users. But by how much? We evaluate how aligned first-token evaluation is with the text output along several dimensions, namely final option choice, refusal rate, choice distribution and robustness under prompt perturbation. Our results show that the two approaches are severely misaligned on all dimensions, reaching mismatch rates over 60%. Models heavily fine-tuned on conversational or safety data are especially impacted. Crucially, models remain misaligned even when we increasingly constrain prompts, i.e., force them to start with an option letter or example template. Our findings i) underscore the importance of inspecting the text output as well and ii) caution against relying solely on first-token evaluation.