Learning Time-Sensitive Strategies in Space Fortress

TL;DR

This paper introduces Space Fortress, a challenging RL environment with reward sparsity and strategy reversals, and demonstrates how tailored enhancements and transfer learning significantly improve deep RL performance.

Contribution

The paper presents new modifications to existing RL algorithms to handle time-sensitive, complex environments like Space Fortress, addressing previously unsolved challenges.

Findings

Standard deep RL algorithms fail on Space Fortress

Enhancements lead to substantial performance improvements

Transfer learning further boosts success rates

Abstract

Although there has been remarkable progress and impressive performance on reinforcement learning (RL) on Atari games, there are many problems with challenging characteristics that have not yet been explored in Deep Learning for RL. These include reward sparsity, abrupt context-dependent reversals of strategy and time-sensitive game play. In this paper, we present Space Fortress, a game that incorporates all these characteristics and experimentally show that the presence of any of these renders state of the art Deep RL algorithms incapable of learning. Then, we present our enhancements to an existing algorithm and show big performance increases through each enhancement through an ablation study. We discuss how each of these enhancements was able to help and also argue that appropriate transfer learning boosts performance.