Exploiting Primacy Effect To Improve Large Language Models

TL;DR

This paper investigates primacy bias in fine-tuned Large Language Models and demonstrates that reordering answer options based on semantic similarity can significantly improve multiple choice question answering accuracy.

Contribution

It reveals that fine-tuning amplifies primacy bias in LLMs and proposes a novel reordering strategy to leverage this bias for better performance.

Findings

Reordering options based on semantic similarity improves MCQA accuracy.

Fine-tuning increases primacy bias in LLMs.

Bias-aware reordering enhances NLP task performance.

Abstract

Large Language Models (LLMs) have become essential in many Natural Language Processing (NLP) tasks, leveraging extensive pre-training and fine-tuning to achieve high accuracy. However, like humans, LLMs exhibit biases, particularly positional biases such as primacy and recency effects, which can influence the accuracy of the answers. The primacy effect-where items presented first are more likely to be remembered or selected-plays a key role in Multiple Choice Question Answering (MCQA), where the order of answer options can affect prediction outcomes. This study focuses on primacy bias in fine-tuned LLMs: We first show that fine-tuning amplifies this bias, probably due to exposure to human-like patterns. Hence, we strategically leverage this effect by reordering response options based on semantic similarity to the query, without requiring knowledge of the correct answer. Our experimental results show that this approach significantly improves performance in MCQA. More generally, our findings underscore the dual nature of biases as both challenges and opportunities, offering insights for bias-aware model design and NLP applications.