Adaptive Risk-Aware Bidding with Budget Constraint in Display Advertising

TL;DR

This paper introduces a novel adaptive risk-aware bidding algorithm for display advertising that incorporates uncertainty and risk preferences of demand-side platforms using reinforcement learning, improving bidding strategies under budget constraints.

Contribution

It is the first to integrate estimation uncertainty and dynamic risk preferences into a reinforcement learning framework for RTB bidding strategies.

Findings

Outperforms state-of-the-art methods on public datasets.

Effectively models risk preferences using VaR.

Demonstrates robustness under budget constraints.

Abstract

Real-time bidding (RTB) has become a major paradigm of display advertising. Each ad impression generated from a user visit is auctioned in real time, where demand-side platform (DSP) automatically provides bid price usually relying on the ad impression value estimation and the optimal bid price determination. However, the current bid strategy overlooks large randomness of the user behaviors (e.g., click) and the cost uncertainty caused by the auction competition. In this work, we explicitly factor in the uncertainty of estimated ad impression values and model the risk preference of a DSP under a specific state and market environment via a sequential decision process. Specifically, we propose a novel adaptive risk-aware bidding algorithm with budget constraint via reinforcement learning, which is the first to simultaneously consider estimation uncertainty and the dynamic risk tendency of a DSP. We theoretically unveil the intrinsic relation between the uncertainty and the risk tendency based on value at risk (VaR). Consequently, we propose two instantiations to model risk tendency, including an expert knowledge-based formulation embracing three essential properties and an adaptive learning method based on self-supervised reinforcement learning. We conduct extensive experiments on public datasets and show that the proposed framework outperforms state-of-the-art methods in practical settings.