LLM Driven Processes to Foster Explainable AI

TL;DR

This paper introduces a modular, explainable AI system that combines large language models with formal decision frameworks to produce auditable, transparent decision support tools, demonstrated through a logistics case study.

Contribution

It presents a novel pipeline integrating LLMs with formal game-theoretic models for explainability and auditability in decision support systems.

Findings

Achieved 55.5% factor alignment with human baseline.

Role agreement over matches was 57%.

LLM judge scored runs comparable to human baseline.

Abstract

We present a modular, explainable LLM-agent pipeline for decision support that externalizes reasoning into auditable artifacts. The system instantiates three frameworks: Vester's Sensitivity Model (factor set, signed impact matrix, systemic roles, feedback loops); normal-form games (strategies, payoff matrix, equilibria); and sequential games (role-conditioned agents, tree construction, backward induction), with swappable modules at every step. LLM components (default: GPT-5) are paired with deterministic analyzers for equilibria and matrix-based role classification, yielding traceable intermediates rather than opaque outputs. In a real-world logistics case (100 runs), mean factor alignment with a human baseline was 55.5\% over 26 factors and 62.9\% on the transport-core subset; role agreement over matches was 57\%. An LLM judge using an eight-criterion rubric (max 100) scored runs on par with a reconstructed human baseline. Configurable LLM pipelines can thus mimic expert workflows with transparent, inspectable steps.