Optimizing Sponsored Search Ranking Strategy by Deep Reinforcement Learning

TL;DR

This paper introduces a deep reinforcement learning framework to optimize sponsored search ranking functions, improving adaptability and performance by combining offline simulation-based training with online data-driven refinement.

Contribution

The work presents a novel deep reinforcement learning approach for sponsored search ranking, integrating offline simulation and online adaptation to enhance ranking effectiveness.

Findings

Effective in large-scale sponsored search platform

Improves ranking performance and adaptability

Combines offline and online learning for optimization

Abstract

Sponsored search is an indispensable business model and a major revenue contributor of almost all the search engines. From the advertisers' side, participating in ranking the search results by paying for the sponsored search advertisement to attract more awareness and purchase facilitates their commercial goal. From the users' side, presenting personalized advertisement reflecting their propensity would make their online search experience more satisfactory. Sponsored search platforms rank the advertisements by a ranking function to determine the list of advertisements to show and the charging price for the advertisers. Hence, it is crucial to find a good ranking function which can simultaneously satisfy the platform, the users and the advertisers. Moreover, advertisements showing positions under different queries from different users may associate with advertisement candidates of different bid price distributions and click probability distributions, which requires the ranking functions to be optimized adaptively to the traffic characteristics. In this work, we proposed a generic framework to optimize the ranking functions by deep reinforcement learning methods. The framework is composed of two parts: an offline learning part which initializes the ranking functions by learning from a simulated advertising environment, allowing adequate exploration of the ranking function parameter space without hurting the performance of the commercial platform. An online learning part which further optimizes the ranking functions by adapting to the online data distribution. Experimental results on a large-scale sponsored search platform confirm the effectiveness of the proposed method.