Hybrid Transfer in Deep Reinforcement Learning for Ads Allocation

TL;DR

This paper introduces SHTAA, a transfer learning method for deep reinforcement learning in ads allocation, improving performance in data-scarce scenarios and boosting platform revenue.

Contribution

It proposes a novel similarity-based hybrid transfer approach for deep RL in ads allocation, effectively transferring knowledge from data-rich to data-poor entrances.

Findings

Enhanced revenue in data-scarce entrances

Improved performance over baseline methods

Effective transfer of knowledge across entrances

Abstract

Ads allocation, which involves allocating ads and organic items to limited slots in feed with the purpose of maximizing platform revenue, has become a research hotspot. Notice that, e-commerce platforms usually have multiple entrances for different categories and some entrances have few visits. Data from these entrances has low coverage, which makes it difficult for the agent to learn. To address this challenge, we propose Similarity-based Hybrid Transfer for Ads Allocation (SHTAA), which effectively transfers samples as well as knowledge from data-rich entrance to data-poor entrance. Specifically, we define an uncertainty-aware similarity for MDP to estimate the similarity of MDP for different entrances. Based on this similarity, we design a hybrid transfer method, including instance transfer and strategy transfer, to efficiently transfer samples and knowledge from one entrance to another. Both offline and online experiments on Meituan food delivery platform demonstrate that the proposed method could achieve better performance for data-poor entrance and increase the revenue for the platform.