MaRCA: Multi-Agent Reinforcement Learning for Dynamic Computation Allocation in Large-Scale Recommender Systems

TL;DR

MaRCA is a multi-agent reinforcement learning framework that optimizes computation resource allocation in large-scale recommender systems, significantly improving revenue while efficiently managing computational costs.

Contribution

It introduces a novel multi-agent RL approach with end-to-end optimization, AutoBucket cost estimation, and MPC-based traffic forecasting for recommender systems.

Findings

Handled hundreds of billions of ad requests daily

Achieved 16.67% revenue uplift

Demonstrated effectiveness in a real-world e-commerce platform

Abstract

Modern recommender systems face significant computational challenges due to growing model complexity and traffic scale, making efficient computation allocation critical for maximizing business revenue. Existing approaches typically simplify multi-stage computation resource allocation, neglecting inter-stage dependencies, thus limiting global optimality. In this paper, we propose MaRCA, a multi-agent reinforcement learning framework for end-to-end computation resource allocation in large-scale recommender systems. MaRCA models the stages of a recommender system as cooperative agents, using Centralized Training with Decentralized Execution (CTDE) to optimize revenue under computation resource constraints. We introduce an AutoBucket TestBench for accurate computation cost estimation, and a Model Predictive Control (MPC)-based Revenue-Cost Balancer to proactively forecast traffic loads and adjust the revenue-cost trade-off accordingly. Since its end-to-end deployment in the advertising pipeline of a leading global e-commerce platform in November 2024, MaRCA has consistently handled hundreds of billions of ad requests per day and has delivered a 16.67% revenue uplift using existing computation resources.