Successor Features for Transfer in Alternating Markov Games

TL;DR

This paper introduces a novel transfer learning algorithm for multi-agent Markov games using successor features, enabling effective policy transfer and improved performance in turn-based games.

Contribution

It proposes the GGPI algorithm that applies successor features to multi-agent games, providing a new method for knowledge transfer across different game tasks.

Findings

GGPI achieves high-reward interactions in experiments.

It enables one-shot policy transfer.

It outperforms baseline algorithms in success rate and path efficiency.

Abstract

This paper explores successor features for knowledge transfer in zero-sum, complete-information, and turn-based games. Prior research in single-agent systems has shown that successor features can provide a ``jump start" for agents when facing new tasks with varying reward structures. However, knowledge transfer in games typically relies on value and equilibrium transfers, which heavily depends on the similarity between tasks. This reliance can lead to failures when the tasks differ significantly. To address this issue, this paper presents an application of successor features to games and presents a novel algorithm called Game Generalized Policy Improvement (GGPI), designed to address Markov games in multi-agent reinforcement learning. The proposed algorithm enables the transfer of learning values and policies across games. An upper bound of the errors for transfer is derived as a function the similarity of the task. Through experiments with a turn-based pursuer-evader game, we demonstrate that the GGPI algorithm can generate high-reward interactions and one-shot policy transfer. When further tested in a wider set of initial conditions, the GGPI algorithm achieves higher success rates with improved path efficiency compared to those of the baseline algorithms.