Reviewing the Reviewer: Graph-Enhanced LLMs for E-commerce Appeal Adjudication

TL;DR

This paper introduces a graph-based reasoning framework with explicit action modeling and conflict detection to improve e-commerce appeal adjudication, significantly enhancing alignment with human judgments over baseline models.

Contribution

It proposes the Evidence-Action-Factor-Decision schema and a conflict-aware graph reasoning method to ground decisions in verifiable operations, reducing hallucinations and improving accuracy.

Findings

Achieved 96.3% alignment in real-world deployment

Improved offline accuracy to 95.8% with knowledge graph augmentation

Enhanced baseline performance from 70.8% to 87.5% with action modeling and RMI

Abstract

Hierarchical review workflows, where a second-tier reviewer (Checker) corrects first-tier (Maker) decisions, generate valuable correction signals that encode why initial judgments failed. However, learning from these signals is hindered by information asymmetry: corrections often depend on verification actions unavailable to Makers or automated systems. We address this challenge by introducing explicit action modeling as an inferential constraint that grounds reasoning in verifiable operations rather than unconstrained text generation. We propose the Evidence-Action-Factor-Decision (EAFD) schema, a minimal representation for adjudication reasoning that prevents hallucination through operational grounding and enables learning from correction signals via explicit conflict modeling. Building on this schema, we develop a conflict-aware graph reasoning framework that: (1) constructs EAFD graphs from historical cases capturing Maker-Checker disagreements, (2) aggregates them into a retrievable knowledge base, and (3) performs top-down deductive reasoning for new cases by projecting validated resolution paths from precedents. A distinctive capability is the Request More Information (RMI) outcome: when evidence is insufficient, the system identifies precisely which verification actions remain unexecuted and generates targeted information requests. We evaluate the framework in large-scale e-commerce seller appeal adjudication. While a standard LLM-only baseline achieves only 70.8% alignment with human experts, incorporating action modeling with RMI improves alignment to 87.5%. Augmenting this with the retrieval-based knowledge graph yields the best offline performance of 95.8%. Following online deployment, the framework maintains robust performance, achieving a 96.3% alignment rate in production, demonstrating its real-world effectiveness.