Investigating the Effects of Cognitive Biases in Prompts on Large Language Model Outputs

TL;DR

This study explores how cognitive biases embedded in prompts influence Large Language Model outputs, revealing that even subtle biases can significantly distort answers and internal decision processes, emphasizing the need for bias-aware prompt design.

Contribution

Introduces a systematic framework to embed and assess cognitive biases in prompts, analyzing their impact on LLM accuracy and internal attention mechanisms across multiple datasets.

Findings

Cognitive biases significantly alter LLM answer choices.

Biases affect internal attention distributions in LLMs.

Bias-aware prompt design can improve LLM robustness.

Abstract

This paper investigates the influence of cognitive biases on Large Language Models (LLMs) outputs. Cognitive biases, such as confirmation and availability biases, can distort user inputs through prompts, potentially leading to unfaithful and misleading outputs from LLMs. Using a systematic framework, our study introduces various cognitive biases into prompts and assesses their impact on LLM accuracy across multiple benchmark datasets, including general and financial Q&A scenarios. The results demonstrate that even subtle biases can significantly alter LLM answer choices, highlighting a critical need for bias-aware prompt design and mitigation strategy. Additionally, our attention weight analysis highlights how these biases can alter the internal decision-making processes of LLMs, affecting the attention distribution in ways that are associated with output inaccuracies. This research has implications for Al developers and users in enhancing the robustness and reliability of Al applications in diverse domains.