Hierarchical principles of embodied reinforcement learning: A review

TL;DR

This review highlights the importance of hierarchical mental representations in biological problem-solving and discusses how current AI methods lack integration of key cognitive mechanisms for advanced problem-solving.

Contribution

It provides a comprehensive survey linking cognitive mechanisms to hierarchical reinforcement learning and identifies gaps in integrating these mechanisms in AI.

Findings

Cognitive mechanisms involve compositional abstraction, curiosity, and forward models.

Existing computational architectures implement these mechanisms independently.

Integrating these mechanisms could enhance AI problem-solving capabilities.

Abstract

Cognitive Psychology and related disciplines have identified several critical mechanisms that enable intelligent biological agents to learn to solve complex problems. There exists pressing evidence that the cognitive mechanisms that enable problem-solving skills in these species build on hierarchical mental representations. Among the most promising computational approaches to provide comparable learning-based problem-solving abilities for artificial agents and robots is hierarchical reinforcement learning. However, so far the existing computational approaches have not been able to equip artificial agents with problem-solving abilities that are comparable to intelligent animals, including human and non-human primates, crows, or octopuses. Here, we first survey the literature in Cognitive Psychology, and related disciplines, and find that many important mental mechanisms involve compositional abstraction, curiosity, and forward models. We then relate these insights with contemporary hierarchical reinforcement learning methods, and identify the key machine intelligence approaches that realise these mechanisms. As our main result, we show that all important cognitive mechanisms have been implemented independently in isolated computational architectures, and there is simply a lack of approaches that integrate them appropriately. We expect our results to guide the development of more sophisticated cognitively inspired hierarchical methods, so that future artificial agents achieve a problem-solving performance on the level of intelligent animals.