What Makes the Preferred Thinking Direction for LLMs in Multiple-choice Questions?

TL;DR

This paper explores alternative text generation orderings in language models, demonstrating that right-to-left training can outperform traditional left-to-right methods on multiple-choice question benchmarks, with insights into underlying factors.

Contribution

It introduces and empirically evaluates right-to-left training for LLMs, revealing its advantages over left-to-right approaches in reasoning tasks and providing theoretical analysis of the factors involved.

Findings

R2L models outperform L2R models on several MCQ benchmarks

Performance differences linked to calibration, computability, and entropy

Controlled experiments with arithmetic tasks disentangle influencing factors

Abstract

Language models usually use left-to-right (L2R) autoregressive factorization. However, L2R factorization may not always be the best inductive bias. Therefore, we investigate whether alternative factorizations of the text distribution could be beneficial in some tasks. We investigate right-to-left (R2L) training as a compelling alternative, focusing on multiple-choice questions (MCQs) as a test bed for knowledge extraction and reasoning. Through extensive experiments across various model sizes (2B-8B parameters) and training datasets, we find that R2L models can significantly outperform L2R models on several MCQ benchmarks, including logical reasoning, commonsense understanding, and truthfulness assessment tasks. Our analysis reveals that this performance difference may be fundamentally linked to multiple factors including calibration, computability, and directional conditional entropy. We analyze the impact of these factors through controlled simulation studies using arithmetic tasks, where the impacting factors can be better disentangled. Our work demonstrates that exploring alternative factorizations of the text distribution can lead to improvements in LLM capabilities and provides theoretical insights into optimal factorization towards approximating human language distribution, and when each reasoning order might be more advantageous. Our code and checkpoints are released at https://github.com/apple/ml-reversal-blessing.