Evolving Hierarchical Memory-Prediction Machines in Multi-Task Reinforcement Learning

TL;DR

This paper presents a method using genetic programming to evolve hierarchical memory-based agents capable of multi-task reinforcement learning without explicit task identifiers, demonstrating success across diverse environments.

Contribution

It introduces a novel approach that evolves hierarchical memory structures enabling agents to generalize across multiple tasks without task-specific inputs.

Findings

Hierarchical memory structures improve multi-task learning performance.

Evolved agents perform competitively with task-specific agents.

Dynamic complexity allows efficient real-time operation.

Abstract

A fundamental aspect of behaviour is the ability to encode salient features of experience in memory and use these memories, in combination with current sensory information, to predict the best action for each situation such that long-term objectives are maximized. The world is highly dynamic, and behavioural agents must generalize across a variety of environments and objectives over time. This scenario can be modeled as a partially-observable multi-task reinforcement learning problem. We use genetic programming to evolve highly-generalized agents capable of operating in six unique environments from the control literature, including OpenAI's entire Classic Control suite. This requires the agent to support discrete and continuous actions simultaneously. No task-identification sensor inputs are provided, thus agents must identify tasks from the dynamics of state variables alone and define control policies for each task. We show that emergent hierarchical structure in the evolving programs leads to multi-task agents that succeed by performing a temporal decomposition and encoding of the problem environments in memory. The resulting agents are competitive with task-specific agents in all six environments. Furthermore, the hierarchical structure of programs allows for dynamic run-time complexity, which results in relatively efficient operation.