Reinforcement Mechanism Design, with Applications to Dynamic Pricing in Sponsored Search Auctions

TL;DR

This paper introduces a data-driven reinforcement learning approach to optimize reserve prices in sponsored search auctions, outperforming existing static and dynamic strategies by learning from real bidding data.

Contribution

It proposes a novel reinforcement mechanism design framework that learns optimal reserve prices through real data and reinforcement learning, moving beyond traditional game-theoretic methods.

Findings

Outperforms static reserve price strategies.

Effective in a GSP-like auction environment.

Demonstrates improved revenue through simulation.

Abstract

In this study, we apply reinforcement learning techniques and propose what we call reinforcement mechanism design to tackle the dynamic pricing problem in sponsored search auctions. In contrast to previous game-theoretical approaches that heavily rely on rationality and common knowledge among the bidders, we take a data-driven approach, and try to learn, over repeated interactions, the set of optimal reserve prices. We implement our approach within the current sponsored search framework of a major search engine: we first train a buyer behavior model, via a real bidding data set, that accurately predicts bids given information that bidders are aware of, including the game parameters disclosed by the search engine, as well as the bidders' KPI data from previous rounds. We then put forward a reinforcement/MDP (Markov Decision Process) based algorithm that optimizes reserve prices over time, in a GSP-like auction. Our simulations demonstrate that our framework outperforms static optimization strategies including the ones that are currently in use, as well as several other dynamic ones.