SupChain-Bench: Benchmarking Large Language Models for Real-World Supply Chain Management

TL;DR

SupChain-Bench is a comprehensive benchmark designed to evaluate large language models' ability to perform reliable, long-horizon supply chain management tasks grounded in domain-specific procedures.

Contribution

The paper introduces SupChain-Bench, a new benchmark for assessing LLMs in supply chain workflows, and proposes SupChain-ReAct, a framework for autonomous tool-based orchestration.

Findings

Substantial gaps in execution reliability across models.

SupChain-ReAct achieves the strongest tool-calling performance.

Highlights significant room for improvement in LLM-based supply chain agents.

Abstract

Large language models (LLMs) have shown promise in complex reasoning and tool-based decision making, motivating their application to real-world supply chain management. However, supply chain workflows require reliable long-horizon, multi-step orchestration grounded in domain-specific procedures, which remains challenging for current models. To systematically evaluate LLM performance in this setting, we introduce SupChain-Bench, a unified real-world benchmark that assesses both supply chain domain knowledge and long-horizon tool-based orchestration grounded in standard operating procedures (SOPs). Our experiments reveal substantial gaps in execution reliability across models. We further propose SupChain-ReAct, an SOP-free framework that autonomously synthesizes executable procedures for tool use, achieving the strongest and most consistent tool-calling performance. Our work establishes a principled benchmark for studying reliable long-horizon orchestration in real-world operational settings and highlights significant room for improvement in LLM-based supply chain agents.