Let the Agent Steer: Closed-Loop Ranking Optimization via Influence Exchange

TL;DR

This paper introduces Sortify, an autonomous LLM-driven ranking optimization agent that dynamically manages influence exchange among factors to improve recommendation system performance in large-scale deployments.

Contribution

It presents a novel influence exchange framework with a dual-channel approach, LLM meta-controller, and persistent memory, enabling fully autonomous online ranking optimization.

Findings

In Country A, GMV increased from -3.6% to +9.2% within 7 rounds.

In Country B, the system achieved +4.15% GMV/UU and +3.58% Ads Revenue in a 7-day A/B test.

Full deployment was achieved after successful cold-start results.

Abstract

Recommendation ranking is fundamentally an influence allocation problem: a sorting formula distributes ranking influence among competing factors, and the business outcome depends on finding the optimal "exchange rates" among them. However, offline proxy metrics systematically misjudge how influence reallocation translates to online impact, with asymmetric bias across metrics that a single calibration factor cannot correct. We present Sortify, the first fully autonomous LLM-driven ranking optimization agent deployed in a large-scale production recommendation system. The agent reframes ranking optimization as continuous influence exchange, closing the full loop from diagnosis to parameter deployment without human intervention. It addresses structural problems through three mechanisms: (1) a dual-channel framework grounded in Savage's Subjective Expected Utility (SEU) that decouples offline-online transfer correction (Belief channel) from constraint penalty adjustment (Preference channel); (2) an LLM meta-controller operating on framework-level parameters rather than low-level search variables; (3) a persistent Memory DB with 7 relational tables for cross-round learning. Its core metric, Influence Share, provides a decomposable measure where all factor contributions sum to exactly 100%. Sortify has been deployed across two markets. In Country A, the agent pushed GMV from -3.6% to +9.2% within 7 rounds with peak orders reaching +12.5%. In Country B, a cold-start deployment achieved +4.15% GMV/UU and +3.58% Ads Revenue in a 7-day A/B test, leading to full production rollout.