Learning to Play Sequential Games versus Unknown Opponents

TL;DR

This paper introduces a kernel-based online learning algorithm for sequential games against unknown opponents, providing regret guarantees and demonstrating effectiveness in traffic routing and conservation tasks.

Contribution

It develops a novel bilevel optimization and online learning method for unknown opponent modeling in sequential games, with theoretical regret bounds and practical applications.

Findings

Regret scales sublinearly with game rounds.

Algorithm performs well against adversarial opponents.

Effective in traffic routing and wildlife conservation tasks.

Abstract

We consider a repeated sequential game between a learner, who plays first, and an opponent who responds to the chosen action. We seek to design strategies for the learner to successfully interact with the opponent. While most previous approaches consider known opponent models, we focus on the setting in which the opponent's model is unknown. To this end, we use kernel-based regularity assumptions to capture and exploit the structure in the opponent's response. We propose a novel algorithm for the learner when playing against an adversarial sequence of opponents. The algorithm combines ideas from bilevel optimization and online learning to effectively balance between exploration (learning about the opponent's model) and exploitation (selecting highly rewarding actions for the learner). Our results include algorithm's regret guarantees that depend on the regularity of the opponent's response and scale sublinearly with the number of game rounds. Moreover, we specialize our approach to repeated Stackelberg games, and empirically demonstrate its effectiveness in a traffic routing and wildlife conservation task