More is More: Addition Bias in Large Language Models

TL;DR

This paper reveals that large language models exhibit a strong bias towards making additive modifications over subtractive ones across various tasks, which could impact their efficiency and resource consumption.

Contribution

The study systematically demonstrates the presence of additive bias in multiple large language models through controlled experiments, highlighting a previously underexplored aspect of LLM behavior.

Findings

LLMs prefer adding over removing in tasks like palindrome creation and tower balancing.

Additive bias leads to longer summaries and increased resource use.

Bias is consistent across different models and tasks.

Abstract

In this paper, we investigate the presence of additive bias in Large Language Models (LLMs), drawing a parallel to the cognitive bias observed in humans where individuals tend to favor additive over subtractive changes. Using a series of controlled experiments, we tested various LLMs, including GPT-3.5 Turbo, Claude 3.5 Sonnet, Mistral, Math$\Sigma$tral, and Llama 3.1, on tasks designed to measure their propensity for additive versus subtractive modifications. Our findings demonstrate a significant preference for additive changes across all tested models. For example, in a palindrome creation task, Llama 3.1 favored adding letters 97.85% of the time over removing them. Similarly, in a Lego tower balancing task, GPT-3.5 Turbo chose to add a brick 76.38% of the time rather than remove one. In a text summarization task, Mistral 7B produced longer summaries in 59.40% to 75.10% of cases when asked to improve its own or others' writing. These results indicate that, similar to humans, LLMs exhibit a marked additive bias, which might have implications when LLMs are used on a large scale. Addittive bias might increase resource use and environmental impact, leading to higher economic costs due to overconsumption and waste. This bias should be considered in the development and application of LLMs to ensure balanced and efficient problem-solving approaches.