Combining imagination and heuristics to learn strategies that generalize

TL;DR

This paper introduces a hierarchical reinforcement learning model that combines heuristics and imagination, inspired by human prefrontal networks, to improve learning speed, transferability, and interpretability in complex environments.

Contribution

It presents a novel stumbler-strategist network that integrates heuristics and imagination, enhancing generalization and learning efficiency in reinforcement learning tasks.

Findings

Accelerates learning in Wythoff's game

Enhances transfer to new games

Improves model interpretability

Abstract

Deep reinforcement learning can match or exceed human performance in stable contexts, but with minor changes to the environment artificial networks, unlike humans, often cannot adapt. Humans rely on a combination of heuristics to simplify computational load and imagination to extend experiential learning to new and more challenging environments. Motivated by theories of the hierarchical organization of the human prefrontal networks, we have developed a model of hierarchical reinforcement learning that combines both heuristics and imagination into a stumbler-strategist network. We test performance of this network using Wythoff's game, a gridworld environment with a known optimal strategy. We show that a heuristic labeling of each position as hot or cold, combined with imagined play, both accelerates learning and promotes transfer to novel games, while also improving model interpretability.