A Reality check of the benefits of LLM in business

TL;DR

This paper evaluates the practical benefits and limitations of large language models in business contexts through experiments on real-world data, providing insights into their readiness for organizational use.

Contribution

It offers the first quantified analysis of LLMs applied to core business operations, highlighting their capabilities and challenges in real-world applications.

Findings

LLMs show potential in business functions like planning and decision-making.

Limitations include bias, poor contextual understanding, and prompt sensitivity.

Experimental results inform future research and organizational deployment strategies.

Abstract

Large language models (LLMs) have achieved remarkable performance in language understanding and generation tasks by leveraging vast amounts of online texts. Unlike conventional models, LLMs can adapt to new domains through prompt engineering without the need for retraining, making them suitable for various business functions, such as strategic planning, project implementation, and data-driven decision-making. However, their limitations in terms of bias, contextual understanding, and sensitivity to prompts raise concerns about their readiness for real-world applications. This paper thoroughly examines the usefulness and readiness of LLMs for business processes. The limitations and capacities of LLMs are evaluated through experiments conducted on four accessible LLMs using real-world data. The findings have significant implications for organizations seeking to leverage generative AI and provide valuable insights into future research directions. To the best of our knowledge, this represents the first quantified study of LLMs applied to core business operations and challenges.