OI-Bench: An Option Injection Benchmark for Evaluating LLM Susceptibility to Directive Interference

TL;DR

This paper introduces OI-Bench, a comprehensive benchmark to evaluate large language models' vulnerability to directive interference through option injection, revealing significant susceptibility and variability across models.

Contribution

The paper presents a novel benchmarking approach and dataset for systematically assessing LLM susceptibility to directive interference via option injection.

Findings

Substantial vulnerabilities in LLMs to directive interference.

Heterogeneous robustness observed across different models.

Evaluation of mitigation strategies shows varied effectiveness.

Abstract

Benchmarking large language models (LLMs) is critical for understanding their capabilities, limitations, and robustness. In addition to interface artifacts, prior studies have shown that LLM decisions can be influenced by directive signals such as social cues, framing, and instructions. In this work, we introduce option injection, a benchmarking approach that augments the multiple-choice question answering (MCQA) interface with an additional option containing a misleading directive, leveraging standardized choice structure and scalable evaluation. We construct OI-Bench, a benchmark of 3,000 questions spanning knowledge, reasoning, and commonsense tasks, with 16 directive types covering social compliance, bonus framing, threat framing, and instructional interference. This setting combines manipulation of the choice interface with directive-based interference, enabling systematic assessment of model susceptibility. We evaluate 12 LLMs to analyze attack success rates, behavioral responses, and further investigate mitigation strategies ranging from inference-time prompting to post-training alignment. Experimental results reveal substantial vulnerabilities and heterogeneous robustness across models. OI-Bench is expected to support more systematic evaluation of LLM robustness to directive interference within choice-based interfaces.