Task-Relevant Object Discovery and Categorization for Playing First-person Shooter Games

TL;DR

This paper introduces an efficient method for object discovery and categorization in FPS games, improving deep reinforcement learning by providing relevant object information as auxiliary input.

Contribution

The method detects and clusters salient frame segments based on optical flow and features, enabling better object representation for reinforcement learning in FPS games.

Findings

Enhanced performance of DRQN with object-based side inputs

Effective categorization of objects relevant to gameplay

Reduced data requirements for training deep RL models

Abstract

We consider the problem of learning to play first-person shooter (FPS) video games using raw screen images as observations and keyboard inputs as actions. The high-dimensionality of the observations in this type of applications leads to prohibitive needs of training data for model-free methods, such as the deep Q-network (DQN), and its recurrent variant DRQN. Thus, recent works focused on learning low-dimensional representations that may reduce the need for data. This paper presents a new and efficient method for learning such representations. Salient segments of consecutive frames are detected from their optical flow, and clustered based on their feature descriptors. The clusters typically correspond to different discovered categories of objects. Segments detected in new frames are then classified based on their nearest clusters. Because only a few categories are relevant to a given task, the importance of a category is defined as the correlation between its occurrence and the agent's performance. The result is encoded as a vector indicating objects that are in the frame and their locations, and used as a side input to DRQN. Experiments on the game Doom provide a good evidence for the benefit of this approach.