AIM-Bench: Evaluating Decision-making Biases of Agentic LLM as Inventory Manager

TL;DR

This paper introduces AIM-Bench, a benchmark to evaluate decision-making biases of LLM agents in inventory management, revealing biases similar to humans and exploring mitigation strategies for supply chain applications.

Contribution

The paper presents AIM-Bench, the first benchmark for assessing LLM decision biases in supply chain scenarios, and investigates bias mitigation strategies like cognitive reflection and information sharing.

Findings

LLMs exhibit human-like decision biases in inventory tasks.

Strategies like cognitive reflection reduce biases.

Bias mitigation improves decision quality in supply chain contexts.

Abstract

Recent advances in mathematical reasoning and the long-term planning capabilities of large language models (LLMs) have precipitated the development of agents, which are being increasingly leveraged in business operations processes. Decision models to optimize inventory levels are one of the core elements of operations management. However, the capabilities of the LLM agent in making inventory decisions in uncertain contexts, as well as the decision-making biases (e.g. framing effect, etc.) of the agent, remain largely unexplored. This prompts concerns regarding the capacity of LLM agents to effectively address real-world problems, as well as the potential implications of biases that may be present. To address this gap, we introduce AIM-Bench, a novel benchmark designed to assess the decision-making behaviour of LLM agents in uncertain supply chain management scenarios through a diverse series of inventory replenishment experiments. Our results reveal that different LLMs typically exhibit varying degrees of decision bias that are similar to those observed in human beings. In addition, we explored strategies to mitigate the pull-to-centre effect and the bullwhip effect, namely cognitive reflection and implementation of information sharing. These findings underscore the need for careful consideration of the potential biases in deploying LLMs in Inventory decision-making scenarios. We hope that these insights will pave the way for mitigating human decision bias and developing human-centred decision support systems for supply chains.