Bid Optimization using Maximum Entropy Reinforcement Learning

TL;DR

This paper introduces a novel reinforcement learning approach using maximum entropy RL to optimize impression-level bidding strategies in real-time bidding, improving efficiency and outperforming baseline methods.

Contribution

It proposes a new method combining linear bidding functions with maximum entropy RL to optimize adjustment factors at the impression level in RTB.

Findings

Outperforms baseline bidding strategies in empirical tests

Demonstrates the effectiveness of maximum entropy RL in dynamic bidding environments

Achieves higher cost efficiency in online advertising

Abstract

Real-time bidding (RTB) has become a critical way of online advertising. In RTB, an advertiser can participate in bidding ad impressions to display its advertisements. The advertiser determines every impression's bidding price according to its bidding strategy. Therefore, a good bidding strategy can help advertisers improve cost efficiency. This paper focuses on optimizing a single advertiser's bidding strategy using reinforcement learning (RL) in RTB. Unfortunately, it is challenging to optimize the bidding strategy through RL at the granularity of impression due to the highly dynamic nature of the RTB environment. In this paper, we first utilize a widely accepted linear bidding function to compute every impression's base price and optimize it by a mutable adjustment factor derived from the RTB auction environment, to avoid optimizing every impression's bidding price directly. Specifically, we use the maximum entropy RL algorithm (Soft Actor-Critic) to optimize the adjustment factor generation policy at the impression-grained level. Finally, the empirical study on a public dataset demonstrates that the proposed bidding strategy has superior performance compared with the baselines.