Playing Atari Ball Games with Hierarchical Reinforcement Learning

TL;DR

This paper explores how incorporating human-like social learning and instructions into hierarchical reinforcement learning can significantly speed up the training process in Atari games like Tennis and Pong.

Contribution

It introduces a method to utilize human instructions for task decomposition and option construction in hierarchical reinforcement learning, enhancing learning efficiency.

Findings

Instructions improve task understanding and decomposition.

Hierarchical approach accelerates learning compared to baseline.

Effective in Atari Tennis and Pong environments.

Abstract

Human beings are particularly good at reasoning and inference from just a few examples. When facing new tasks, humans will leverage knowledge and skills learned before, and quickly integrate them with the new task. In addition to learning by experimentation, human also learn socio-culturally through instructions and learning by example. In this way humans can learn much faster compared with most current artificial intelligence algorithms in many tasks. In this paper, we test the idea of speeding up machine learning through social learning. We argue that in solving real-world problems, especially when the task is designed by humans, and/or for humans, there are typically instructions from user manuals and/or human experts which give guidelines on how to better accomplish the tasks. We argue that these instructions have tremendous value in designing a reinforcement learning system which can learn in human fashion, and we test the idea by playing the Atari games Tennis and Pong. We experimentally demonstrate that the instructions provide key information about the task, which can be used to decompose the learning task into sub-systems and construct options for the temporally extended planning, and dramatically accelerate the learning process.