Multitask Neuroevolution for Reinforcement Learning with Long and Short Episodes

TL;DR

This paper introduces NuEMT, a neuroevolutionary multitasking algorithm that improves reinforcement learning efficiency by transferring skills from short to long episodes using importance sampling and adaptive resource allocation.

Contribution

The paper presents the first multitask skills transfer mechanism for neuroevolution in RL, addressing high sample complexity in long-horizon tasks.

Findings

NuEMT reduces data requirements in RL tasks.

It outperforms recent ES baselines in continuous control.

Uses importance sampling for skill transfer.

Abstract

Studies have shown evolution strategies (ES) to be a promising approach for reinforcement learning (RL) with deep neural networks. However, the issue of high sample complexity persists in applications of ES to deep RL over long horizons. This paper is the first to address the shortcoming of today's methods via a novel neuroevolutionary multitasking (NuEMT) algorithm, designed to transfer information from a set of auxiliary tasks (of short episode length) to the target (full length) RL task at hand. The auxiliary tasks, extracted from the target, allow an agent to update and quickly evaluate policies on shorter time horizons. The evolved skills are then transferred to guide the longer and harder task towards an optimal policy. We demonstrate that the NuEMT algorithm achieves data-efficient evolutionary RL, reducing expensive agent-environment interaction data requirements. Our key algorithmic contribution in this setting is to introduce, for the first time, a multitask skills transfer mechanism based on the statistical importance sampling technique. In addition, an adaptive resource allocation strategy is utilized to assign computational resources to auxiliary tasks based on their gleaned usefulness. Experiments on a range of continuous control tasks from the OpenAI Gym confirm that our proposed algorithm is efficient compared to recent ES baselines.