CSCBench: A PVC Diagnostic Benchmark for Commodity Supply Chain Reasoning

TL;DR

This paper introduces CSCBench, a comprehensive benchmark for evaluating large language models' reasoning abilities in commodity supply chain contexts, emphasizing process, rules, and cognitive complexity.

Contribution

The paper presents CSCBench, a novel 2,300+ item benchmark for supply chain reasoning, grounded in a new evaluation framework covering process, commodity rules, and cognitive skills.

Findings

LLMs perform well on process and cognition axes.

Significant performance drop on the variety axis, especially freight agreements.

CSCBench serves as a diagnostic tool for supply chain reasoning capabilities.

Abstract

Large Language Models (LLMs) have achieved remarkable success in general benchmarks, yet their competence in commodity supply chains (CSCs) -- a domain governed by institutional rule systems and feasibility constraints -- remains under-explored. CSC decisions are shaped jointly by process stages (e.g., planning, procurement, delivery), variety-specific rules (e.g., contract specifications and delivery grades), and reasoning depth (from retrieval to multi-step analysis and decision selection). We introduce CSCBench, a 2.3K+ single-choice benchmark for CSC reasoning, instantiated through our PVC 3D Evaluation Framework (Process, Variety, and Cognition). The Process axis aligns tasks with SCOR+Enable; the Variety axis operationalizes commodity-specific rule systems under coupled material-information-financial constraints, grounded in authoritative exchange guidebooks/rulebooks and industry reports; and the Cognition axis follows Bloom's revised taxonomy. Evaluating representative LLMs under a direct prompting setting, we observe strong performance on the Process and Cognition axes but substantial degradation on the Variety axis, especially on Freight Agreements. CSCBench provides a diagnostic yardstick for measuring and improving LLM capabilities in this high-stakes domain.