Serial Position Effects of Large Language Models

TL;DR

This paper investigates serial position effects in Large Language Models, revealing their widespread presence, variability, and the limited effectiveness of prompt-based mitigation, emphasizing the importance of addressing these biases in LLM applications.

Contribution

It provides extensive empirical evidence of serial position effects in LLMs and evaluates prompt-based mitigation strategies, highlighting their inconsistent effectiveness.

Findings

Serial position effects are common across various LLMs and tasks.

Prompt engineering can reduce but not eliminate these biases.

Biases significantly impact inference in unlabeled scenarios.

Abstract

Large Language Models (LLMs) have shown remarkable capabilities in zero-shot learning applications, generating responses to queries using only pre-training information without the need for additional fine-tuning. This represents a significant departure from traditional machine learning approaches. Previous research has indicated that LLMs may exhibit serial position effects, such as primacy and recency biases, which are well-documented cognitive biases in human psychology. Our extensive testing across various tasks and models confirms the widespread occurrence of these effects, although their intensity varies. We also discovered that while carefully designed prompts can somewhat mitigate these biases, their effectiveness is inconsistent. These findings underscore the significance of serial position effects during the inference process, particularly in scenarios where there are no ground truth labels, highlighting the need for greater focus on addressing these effects in LLM applications.