Do Large Language Models Plan Answer Positions? Position Bias in Multiple-Choice Question Generation

TL;DR

This paper investigates how large language models exhibit systematic position biases when generating multiple-choice questions, revealing implicit planning and demonstrating control over answer positioning.

Contribution

It uncovers structured position biases in LLM-generated MCQs, links these biases to internal representations, and introduces activation steering to influence answer placement.

Findings

LLMs show consistent position biases across different models and tasks.

Hidden representations encode signals predicting answer positions.

Activation steering can manipulate and partially control answer placement.

Abstract

Large language models (LLMs) are increasingly used to generate multiple-choice questions (MCQs), where correct answers should ideally be uniformly distributed across options. However, we observe that LLMs exhibit systematic position biases during generation. Through extensive experiments with 10 LLMs and 5 vision-language models (VLMs) on three MCQ generation tasks, we show that these biases are structured, with similar patterns emerging within model families. To investigate the underlying mechanisms, we conduct probing experiments and find that hidden representations in the question stem encode predictive signals of the correct answer position, suggesting that answer position may be implicitly planned during generation. Building on this insight, we apply activation steering to manipulate internal representations and influence answer position. Our results show that steering can partially control positional preferences and substantially shift answer position distributions. Our findings provide a practical framework for studying implicit positional planning in LLMs and highlight the importance of controllable generation for reliable MCQ construction and evaluation.