Offline Reinforcement Learning for Mobile Notifications

TL;DR

This paper introduces an offline reinforcement learning framework for mobile notification systems, improving long-term user engagement by optimizing sequential decisions through offline policy evaluation and online deployment.

Contribution

It presents a novel offline RL approach with a state-marginalized importance sampling method for notification systems, enabling effective offline policy evaluation and deployment.

Findings

Offline RL outperforms traditional response-prediction models.

The proposed evaluation method accurately predicts policy performance.

Online deployment of the RL policy increased user engagement.

Abstract

Mobile notification systems have taken a major role in driving and maintaining user engagement for online platforms. They are interesting recommender systems to machine learning practitioners with more sequential and long-term feedback considerations. Most machine learning applications in notification systems are built around response-prediction models, trying to attribute both short-term impact and long-term impact to a notification decision. However, a user's experience depends on a sequence of notifications and attributing impact to a single notification is not always accurate, if not impossible. In this paper, we argue that reinforcement learning is a better framework for notification systems in terms of performance and iteration speed. We propose an offline reinforcement learning framework to optimize sequential notification decisions for driving user engagement. We describe a state-marginalized importance sampling policy evaluation approach, which can be used to evaluate the policy offline and tune learning hyperparameters. Through simulations that approximate the notifications ecosystem, we demonstrate the performance and benefits of the offline evaluation approach as a part of the reinforcement learning modeling approach. Finally, we collect data through online exploration in the production system, train an offline Double Deep Q-Network and launch a successful policy online. We also discuss the practical considerations and results obtained by deploying these policies for a large-scale recommendation system use-case.