Doc2AHP: Inferring Structured Multi-Criteria Decision Models via Semantic Trees with LLMs

TL;DR

Doc2AHP leverages Large Language Models guided by Analytic Hierarchy Process principles to automatically construct structured decision models from documents, ensuring logical consistency without extensive manual effort.

Contribution

This work introduces Doc2AHP, a novel framework that combines LLMs with AHP constraints to infer structured decision models without annotated data or manual intervention.

Findings

Outperforms baseline models in logical completeness

Enables non-experts to build decision models effectively

Ensures numerical consistency in weight allocation

Abstract

While Large Language Models (LLMs) demonstrate remarkable proficiency in semantic understanding, they often struggle to ensure structural consistency and reasoning reliability in complex decision-making tasks that demand rigorous logic. Although classical decision theories, such as the Analytic Hierarchy Process (AHP), offer systematic rational frameworks, their construction relies heavily on labor-intensive domain expertise, creating an "expert bottleneck" that hinders scalability in general scenarios. To bridge the gap between the generalization capabilities of LLMs and the rigor of decision theory, we propose Doc2AHP, a novel structured inference framework guided by AHP principles. Eliminating the need for extensive annotated data or manual intervention, our approach leverages the structural principles of AHP as constraints to direct the LLM in a constrained search within the unstructured document space, thereby enforcing the logical entailment between parent and child nodes. Furthermore, we introduce a multi-agent weighting mechanism coupled with an adaptive consistency optimization strategy to ensure the numerical consistency of weight allocation. Empirical results demonstrate that Doc2AHP not only empowers non-expert users to construct high-quality decision models from scratch but also significantly outperforms direct generative baselines in both logical completeness and downstream task accuracy.