Thinker: Learning to Plan and Act

TL;DR

Thinker is a reinforcement learning algorithm that enables agents to autonomously learn to plan using a learned world model, achieving state-of-the-art results in complex environments like Sokoban and Atari.

Contribution

It introduces a novel planning approach where RL agents learn to interact with a world model, eliminating the need for handcrafted planning algorithms.

Findings

Achieves state-of-the-art in Sokoban

Demonstrates effective planning in Atari games

Agents learn to interpret and visualize their plans

Abstract

We propose the Thinker algorithm, a novel approach that enables reinforcement learning agents to autonomously interact with and utilize a learned world model. The Thinker algorithm wraps the environment with a world model and introduces new actions designed for interacting with the world model. These model-interaction actions enable agents to perform planning by proposing alternative plans to the world model before selecting a final action to execute in the environment. This approach eliminates the need for handcrafted planning algorithms by enabling the agent to learn how to plan autonomously and allows for easy interpretation of the agent's plan with visualization. We demonstrate the algorithm's effectiveness through experimental results in the game of Sokoban and the Atari 2600 benchmark, where the Thinker algorithm achieves state-of-the-art performance and competitive results, respectively. Visualizations of agents trained with the Thinker algorithm demonstrate that they have learned to plan effectively with the world model to select better actions. Thinker is the first work showing that an RL agent can learn to plan with a learned world model in complex environments.