Learning to Play Air Hockey with Model-Based Deep Reinforcement Learning

TL;DR

This paper explores using model-based deep reinforcement learning with self-play to train autonomous agents for air hockey, emphasizing the importance of generalization, imagination horizon, and handling partial observability.

Contribution

It introduces a novel application of model-based deep RL with self-play for air hockey, addressing overfitting and environment stochasticity.

Findings

Self-play improves generalization to unseen opponents.

Longer imagination horizons lead to more stable learning.

Agents trained with these methods achieve competitive performance.

Abstract

In the context of addressing the Robot Air Hockey Challenge 2023, we investigate the applicability of model-based deep reinforcement learning to acquire a policy capable of autonomously playing air hockey. Our agents learn solely from sparse rewards while incorporating self-play to iteratively refine their behaviour over time. The robotic manipulator is interfaced using continuous high-level actions for position-based control in the Cartesian plane while having partial observability of the environment with stochastic transitions. We demonstrate that agents are prone to overfitting when trained solely against a single playstyle, highlighting the importance of self-play for generalization to novel strategies of unseen opponents. Furthermore, the impact of the imagination horizon is explored in the competitive setting of the highly dynamic game of air hockey, with longer horizons resulting in more stable learning and better overall performance.