Inclusion-of-Thoughts: Mitigating Preference Instability via Purifying the Decision Space

TL;DR

The paper introduces Inclusion-of-Thoughts (IoT), a self-filtering strategy that improves large language models' stability and accuracy in multiple-choice questions by focusing on plausible options and enhancing interpretability.

Contribution

IoT is a novel self-filtering approach that reconstructs MCQs with plausible options, reducing distractor influence and improving reasoning stability and transparency.

Findings

IoT significantly improves chain-of-thought performance on various benchmarks.

The method enhances model interpretability and decision transparency.

Minimal additional computational overhead is required.

Abstract

Multiple-choice questions (MCQs) are widely used to evaluate large language models (LLMs). However, LLMs remain vulnerable to the presence of plausible distractors. This often diverts attention toward irrelevant choices, resulting in unstable oscillation between correct and incorrect answers. In this paper, we propose Inclusion-of-Thoughts (IoT), a progressive self-filtering strategy that is designed to mitigate this cognitive load (i.e., instability of model preferences under the presence of distractors) and enable the model to focus more effectively on plausible answers. Our method operates to reconstruct the MCQ using only plausible option choices, providing a controlled setting for examining comparative judgements and therefore the stability of the model's internal reasoning under perturbation. By explicitly documenting this filtering process, IoT also enhances the transparency and interpretability of the model's decision-making. Extensive empirical evaluation demonstrates that IoT substantially boosts chain-of-thought performance across a range of arithmetic, commonsense reasoning, and educational benchmarks with minimal computational overhead.