Learning model-based strategies in simple environments with hierarchical q-networks

TL;DR

This paper introduces a Hierarchical Q-Network inspired by human brain organization that learns generalizable strategies in simple environments, outperforming traditional RL methods and enabling transparent inspection of learned rules.

Contribution

The novel Hierarchical Q-Network (HQN) demonstrates improved strategy generalization and interpretability in simple environments, inspired by biological hierarchical structures.

Findings

HQN learned heuristic strategies that generalized across game variations.

Traditional RL approaches failed to perform well on Wythoff's game variants.

HQN's internal model of the game was transparent after training.

Abstract

Recent advances in deep learning have allowed artificial agents to rival human-level performance on a wide range of complex tasks; however, the ability of these networks to learn generalizable strategies remains a pressing challenge. This critical limitation is due in part to two factors: the opaque information representation in deep neural networks and the complexity of the task environments in which they are typically deployed. Here we propose a novel Hierarchical Q-Network (HQN) motivated by theories of the hierarchical organization of the human prefrontal cortex, that attempts to identify lower dimensional patterns in the value landscape that can be exploited to construct an internal model of rules in simple environments. We draw on combinatorial games, where there exists a single optimal strategy for winning that generalizes across other features of the game, to probe the strategy generalization of the HQN and other reinforcement learning (RL) agents using variations of Wythoff's game. Traditional RL approaches failed to reach satisfactory performance on variants of Wythoff's Game; however, the HQN learned heuristic-like strategies that generalized across changes in board configuration. More importantly, the HQN allowed for transparent inspection of the agent's internal model of the game following training. Our results show how a biologically inspired hierarchical learner can facilitate learning abstract rules to promote robust and flexible action policies in simplified training environments with clearly delineated optimal strategies.