Reinforcement Learning-based Product Delivery Frequency Control

TL;DR

This paper introduces a reinforcement learning approach for optimizing product recommendation delivery frequency, balancing long-term business value with resource efficiency, demonstrated at an industrial scale.

Contribution

It presents a novel personalized RL-based methodology combined with a robust volume control technique for frequency management in recommender systems.

Findings

Significant improvement in daily metrics and resource efficiency.

First deep RL application for frequency control at industrial scale.

Effective volume control enhances long-term value optimization.

Abstract

Frequency control is an important problem in modern recommender systems. It dictates the delivery frequency of recommendations to maintain product quality and efficiency. For example, the frequency of delivering promotional notifications impacts daily metrics as well as the infrastructure resource consumption (e.g. CPU and memory usage). There remain open questions on what objective we should optimize to represent business values in the long term best, and how we should balance between daily metrics and resource consumption in a dynamically fluctuating environment. We propose a personalized methodology for the frequency control problem, which combines long-term value optimization using reinforcement learning (RL) with a robust volume control technique we termed "Effective Factor". We demonstrate statistically significant improvement in daily metrics and resource efficiency by our method in several notification applications at a scale of billions of users. To our best knowledge, our study represents the first deep RL application on the frequency control problem at such an industrial scale.