Techniques Toward Optimizing Viewability in RTB Ad Campaigns Using Reinforcement Learning

TL;DR

This paper explores the application of reinforcement learning to optimize viewability in real-time bidding ad campaigns, including experiments, simulations, and live tests comparing RL agents to rule-based methods.

Contribution

It presents a comprehensive survey of RL techniques for RTB ad bidding, including experimental results, hyperparameter optimization, and live traffic testing.

Findings

RL agents outperform rule-based approaches in viewability metrics

Simulated training on edge cases improves RL agent robustness

Bayesian optimization enhances hyperparameter tuning for RL algorithms

Abstract

Reinforcement learning (RL) is an effective technique for training decision-making agents through interactions with their environment. The advent of deep learning has been associated with highly notable successes with sequential decision making problems - such as defeating some of the highest-ranked human players at Go. In digital advertising, real-time bidding (RTB) is a common method of allocating advertising inventory through real-time auctions. Bidding strategies need to incorporate logic for dynamically adjusting parameters in order to deliver pre-assigned campaign goals. Here we discuss techniques toward using RL to train bidding agents. As a campaign metric we particularly focused on viewability: the percentage of inventory which goes on to be viewed by an end user. This paper is presented as a survey of techniques and experiments which we developed through the course of this research. We discuss expanding our training data to include edge cases by training on simulated interactions. We discuss the experimental results comparing the performance of several promising RL algorithms, and an approach to hyperparameter optimization of an actor/critic training pipeline through Bayesian optimization. Finally, we present live-traffic tests of some of our RL agents against a rule-based feedback-control approach, demonstrating the potential for this method as well as areas for further improvement. This paper therefore presents an arrangement of our findings in this quickly developing field, and ways that it can be applied to an RTB use case.