Balancing Rigor and Utility: Mitigating Cognitive Biases in Large Language Models for Multiple-Choice Questions

TL;DR

This paper explores how intentionally balancing cognitive biases in large language models can improve decision-making accuracy and efficiency in multiple-choice tasks by using heuristic moderation and abstention strategies.

Contribution

It introduces a novel approach to leverage cognitive biases in LLMs, challenging the idea of eliminating all biases for better decision-making.

Findings

Reducing error rates with heuristic moderation and abstention.

Aligning LLM decisions with human reasoning improves reliability.

Targeted bias inspection enhances decision accuracy.

Abstract

This paper examines the role of cognitive biases in the decision-making processes of large language models (LLMs), challenging the conventional goal of eliminating all biases. When properly balanced, we show that certain cognitive biases can enhance decision-making efficiency through rational deviations and heuristic shortcuts. By introducing heuristic moderation and an abstention option, which allows LLMs to withhold responses when uncertain, we reduce error rates, improve decision accuracy, and optimize decision rates. Using the Balance Rigor and Utility (BRU) dataset, developed through expert collaboration, our findings demonstrate that targeted inspection of cognitive biases aligns LLM decisions more closely with human reasoning, enhancing reliability and suggesting strategies for future improvements. This approach offers a novel way to leverage cognitive biases to improve the practical utility of LLMs across various applications.