Combined Model for Partially-Observable and Non-Observable Task Switching: Solving Hierarchical Reinforcement Learning Problems Statically and Dynamically with Transfer Learning

TL;DR

This paper introduces PONOWMtk, a novel model combining abstract task representations and input storage to effectively handle partially-observable and non-observable tasks in hierarchical reinforcement learning, with static and dynamic transfer learning.

Contribution

The paper presents PONOWMtk, a new integrated model that unifies approaches for PO and NO tasks, advancing the capabilities of the Working Memory Toolkit in autonomous systems.

Findings

PONOWMtk performs well on PO, NO, and combined tasks.

The model effectively adapts to static and dynamic transfer learning scenarios.

Experimental results validate the model's versatility and robustness.

Abstract

An integral function of fully autonomous robots and humans is the ability to focus attention on a few relevant percepts to reach a certain goal while disregarding irrelevant percepts. Humans and animals rely on the interactions between the Pre-Frontal Cortex (PFC) and the Basal Ganglia (BG) to achieve this focus called Working Memory (WM). The Working Memory Toolkit (WMtk) was developed based on a computational neuroscience model of this phenomenon with Temporal Difference (TD) Learning for autonomous systems. Recent adaptations of the toolkit either utilize Abstract Task Representations (ATRs) to solve Non-Observable (NO) tasks or storage of past input features to solve Partially-Observable (PO) tasks, but not both. We propose a new model, PONOWMtk, which combines both approaches, ATRs and input storage, with a static or dynamic number of ATRs. The results of our experiments show that PONOWMtk performs effectively for tasks that exhibit PO, NO, or both properties.