Micro-Objective Learning : Accelerating Deep Reinforcement Learning through the Discovery of Continuous Subgoals

TL;DR

Micro-objective learning (MOL) enhances deep reinforcement learning by estimating state importance and providing additional rewards, significantly improving performance in sparse reward environments like Atari games.

Contribution

MOL introduces a scalable method to discover and utilize continuous subgoals by estimating state importance, accelerating learning in large state spaces.

Findings

MOL doubled the performance in Montezuma's Revenge compared to previous models.

MOL significantly improved scores in Atari games with sparse rewards.

The approach effectively balances exploration and exploitation in complex environments.

Abstract

Recently, reinforcement learning has been successfully applied to the logical game of Go, various Atari games, and even a 3D game, Labyrinth, though it continues to have problems in sparse reward settings. It is difficult to explore, but also difficult to exploit, a small number of successes when learning policy. To solve this issue, the subgoal and option framework have been proposed. However, discovering subgoals online is too expensive to be used to learn options in large state spaces. We propose Micro-objective learning (MOL) to solve this problem. The main idea is to estimate how important a state is while training and to give an additional reward proportional to its importance. We evaluated our algorithm in two Atari games: Montezuma's Revenge and Seaquest. With three experiments to each game, MOL significantly improved the baseline scores. Especially in Montezuma's Revenge, MOL achieved two times better results than the previous state-of-the-art model.