A Multi-Dimensional Quality Scoring Framework for Decentralized LLM Inference with Proof of Quality

TL;DR

This paper introduces a multi-dimensional framework for assessing the quality of decentralized LLM inference outputs, improving reward allocation and robustness against adversarial behaviors.

Contribution

It proposes a modular quality scoring system that decomposes output quality into multiple dimensions, enabling better calibration and integration into incentive mechanisms.

Findings

Reveals task-dependent reliability of quality dimensions.

Calibrated composite scores outperform single evaluators.

Enhances reward mechanisms with robust aggregation and trust weighting.

Abstract

Decentralized large language model (LLM) inference networks can pool heterogeneous compute to scale serving, but they require lightweight and incentive-compatible mechanisms to assess output quality. Prior work introduced cost-aware Proof of Quality (PoQ) and adaptive robust PoQ to allocate rewards under evaluator heterogeneity and adversarial behavior. In this paper, we focus on the quality signal itself and propose a multi-dimensional quality scoring framework that decomposes output quality into modular dimensions, including model and cost priors, structure quality, semantic quality, query-output alignment, and agreement/uncertainty. Using logged outputs from QA and summarization tasks, we systematically audit dimension reliability and show that seemingly reasonable dimensions can be task-dependent and even negatively correlated with reference quality without calibration. While the default composite underperforms a strong single semantic evaluator, ablations reveal that removing unreliable dimensions and re-normalizing weights yields a calibrated composite that matches or exceeds the best single- evaluator and consensus baselines. Finally, we integrate the composite score as a drop-in quality signal in PoQ and demonstrate complementary benefits with robust aggregation and adaptive trust weighting under adversarial evaluator attacks.