Improving LLM First-Token Predictions in Multiple-Choice Question Answering via Output Prefilling

TL;DR

This paper introduces a prefilling technique to improve the accuracy and reliability of first-token probability methods in multiple-choice question answering with large language models, without modifying model parameters.

Contribution

It repurposes a structured prefix to steer models towards valid answers, significantly enhancing FTP-based evaluation performance across various benchmarks.

Findings

Prefilling improves accuracy, calibration, and consistency of FTP methods.

The approach outperforms standard FTP and rivals full decoding methods.

Prefilling is simple, robust, and low-cost for enhancing multiple-choice LLM evaluation.

Abstract

Large Language Models (LLMs) are increasingly evaluated on multiple-choice question answering (MCQA) tasks using *first-token probability* (FTP), which selects the answer option whose initial token has the highest likelihood. While efficient, FTP can be fragile: models may assign high probability to unrelated tokens (*misalignment*) or use a valid token merely as part of a generic preamble rather than as a clear answer choice (*misinterpretation*), undermining the reliability of symbolic evaluation. We propose a simple solution: the *prefilling attack*, a structured natural-language prefix (e.g., "*The correct option is:*") prepended to the model output. Originally explored in AI safety, we repurpose prefilling to steer the model to respond with a clean, valid option, without modifying its parameters. Empirically, the FTP with prefilling strategy substantially improves accuracy, calibration, and output consistency across a broad set of LLMs and MCQA benchmarks. It outperforms standard FTP and often matches the performance of open-ended generation approaches that require full decoding and external classifiers, while being significantly more efficient. Our findings suggest that prefilling is a simple, robust, and low-cost method to enhance the reliability of FTP-based evaluation in multiple-choice settings.