Beyond Words: How Large Language Models Perform in Quantitative Management Problem-Solving

TL;DR

This study evaluates how large language models perform on complex quantitative management problems in a zero-shot setting, revealing strengths in handling multi-step tasks but limitations in accuracy and consistency across models.

Contribution

It provides a comprehensive analysis of LLM capabilities in quantitative decision tasks, highlighting factors affecting performance and comparing multiple models in diverse scenarios.

Findings

28.8% of responses were exactly correct

Scenario complexity significantly degraded accuracy

Performance was stable across repeated queries

Abstract

This study examines how Large Language Models (LLMs) perform when tackling quantitative management decision problems in a zero-shot setting. Drawing on 900 responses generated by five leading models across 20 diverse managerial scenarios, our analysis explores whether these base models can deliver accurate numerical decisions under varying presentation formats, scenario complexities, and repeated attempts. Contrary to prior findings, we observed no significant effects of text presentation format (direct, narrative, or tabular) or text length on accuracy. However, scenario complexity -- particularly in terms of constraints and irrelevant parameters -- strongly influenced performance, often degrading accuracy. Surprisingly, the models handled tasks requiring multiple solution steps more effectively than expected. Notably, only 28.8\% of responses were exactly correct, highlighting limitations in precision. We further found no significant ``learning effect'' across iterations: performance remained stable across repeated queries. Nonetheless, significant variations emerged among the five tested LLMs, with some showing superior binary accuracy. Overall, these findings underscore both the promise and the pitfalls of harnessing LLMs for complex quantitative decision-making, informing managers and researchers about optimal deployment strategies.