Applying Opponent Modeling for Automatic Bidding in Online Repeated Auctions

TL;DR

This paper introduces a reinforcement learning framework with opponent modeling for automatic bidding in repeated online auctions, enabling bidders to learn optimal strategies and converge to equilibrium.

Contribution

It proposes Bid Net and the PG algorithm for strategic bidding, improving utility and enabling convergence to equilibrium in multiagent auction environments.

Findings

PG algorithm maximizes bidder utility in diverse settings

Bidders learn best responses to static opponents

System converges to auction game equilibrium

Abstract

Online auction scenarios, such as bidding searches on advertising platforms, often require bidders to participate repeatedly in auctions for identical or similar items. Most previous studies have only considered the process by which the seller learns the prior-dependent optimal mechanism in a repeated auction. However, in this paper, we define a multiagent reinforcement learning environment in which strategic bidders and the seller learn their strategies simultaneously and design an automatic bidding algorithm that updates the strategy of bidders through online interactions. We propose Bid Net to replace the linear shading function as a representation of the strategic bidders' strategy, which effectively improves the utility of strategy learned by bidders. We apply and revise the opponent modeling methods to design the PG (pseudo-gradient) algorithm, which allows bidders to learn optimal bidding strategies with predictions of the other agents' strategy transition. We prove that when a bidder uses the PG algorithm, it can learn the best response to static opponents. When all bidders adopt the PG algorithm, the system will converge to the equilibrium of the game induced by the auction. In experiments with diverse environmental settings and varying opponent strategies, the PG algorithm maximizes the utility of bidders. We hope that this article will inspire research on automatic bidding strategies for strategic bidders.