Does DQN really learn? Exploring adversarial training schemes in Pong

TL;DR

This paper investigates the effectiveness of adversarial self-play training schemes, Chainer and Pool, in enhancing Atari Pong agents' robustness and internal representations compared to standard DQN training.

Contribution

It introduces and evaluates two novel self-play training schemes, demonstrating improved robustness and richer internal representations in Pong agents.

Findings

Chainer and Pool improve agent robustness against adversarial strategies.

Agents trained with these schemes outperform standard DQN in robustness metrics.

Training with these methods results in richer network activations and better internal feature prediction.

Abstract

In this work, we study two self-play training schemes, Chainer and Pool, and show they lead to improved agent performance in Atari Pong compared to a standard DQN agent -- trained against the built-in Atari opponent. To measure agent performance, we define a robustness metric that captures how difficult it is to learn a strategy that beats the agent's learned policy. Through playing past versions of themselves, Chainer and Pool are able to target weaknesses in their policies and improve their resistance to attack. Agents trained using these methods score well on our robustness metric and can easily defeat the standard DQN agent. We conclude by using linear probing to illuminate what internal structures the different agents develop to play the game. We show that training agents with Chainer or Pool leads to richer network activations with greater predictive power to estimate critical game-state features compared to the standard DQN agent.