Transforming User Defined Criteria into Explainable Indicators with an Integrated LLM AHP System

TL;DR

This paper introduces an interpretable framework that combines large language models with the Analytic Hierarchy Process to convert user criteria into explainable, quantitative indicators for complex text evaluation, improving transparency and efficiency.

Contribution

The paper presents a novel integrated LLM-AHP system that enhances explainability and operational efficiency in text evaluation by combining LLM scoring with AHP-based weighting.

Findings

Achieves high explainability in text scoring tasks.

Maintains comparable predictive power to traditional methods.

Operates efficiently for real-time web applications.

Abstract

Evaluating complex texts across domains requires converting user defined criteria into quantitative, explainable indicators, which is a persistent challenge in search and recommendation systems. Single prompt LLM evaluations suffer from complexity and latency issues, while criterion specific decomposition approaches rely on naive averaging or opaque black-box aggregation methods. We present an interpretable aggregation framework combining LLM scoring with the Analytic Hierarchy Process. Our method generates criterion specific scores via LLM as judge, measures discriminative power using Jensen Shannon distance, and derives statistically grounded weights through AHP pairwise comparison matrices. Experiments on Amazon review quality assessment and depression related text scoring demonstrate that our approach achieves high explainability and operational efficiency while maintaining comparable predictive power, making it suitable for real time latency sensitive web services.