Strategic Attentive Writer for Learning Macro-Actions

TL;DR

The paper introduces STRAW, a neural network architecture that learns to create and update internal plans and macro-actions in reinforcement learning, improving performance on Atari games and demonstrating versatility on sequence data like text.

Contribution

STRAW is a novel neural network that learns high-level macro-actions and internal plans end-to-end without prior knowledge, enhancing exploration and planning in RL.

Findings

Improves performance on Atari games like Ms. Pacman and Frostbite.

Learns to predict frequent n-grams in text prediction tasks.

Capable of learning temporally extended strategies from data.

Abstract

We present a novel deep recurrent neural network architecture that learns to build implicit plans in an end-to-end manner by purely interacting with an environment in reinforcement learning setting. The network builds an internal plan, which is continuously updated upon observation of the next input from the environment. It can also partition this internal representation into contiguous sub- sequences by learning for how long the plan can be committed to - i.e. followed without re-planing. Combining these properties, the proposed model, dubbed STRategic Attentive Writer (STRAW) can learn high-level, temporally abstracted macro- actions of varying lengths that are solely learnt from data without any prior information. These macro-actions enable both structured exploration and economic computation. We experimentally demonstrate that STRAW delivers strong improvements on several ATARI games by employing temporally extended planning strategies (e.g. Ms. Pacman and Frostbite). It is at the same time a general algorithm that can be applied on any sequence data. To that end, we also show that when trained on text prediction task, STRAW naturally predicts frequent n-grams (instead of macro-actions), demonstrating the generality of the approach.