Instance Weighted Incremental Evolution Strategies for Reinforcement Learning in Dynamic Environments

TL;DR

This paper introduces IW-IES, a scalable evolution strategy that adaptively updates policies in dynamic environments by weighting instances based on novelty and quality, enabling rapid learning adaptation.

Contribution

It proposes a novel incremental learning method for evolution strategies in dynamic environments, incorporating instance weighting to improve adaptation and scalability.

Findings

IW-IES outperforms existing methods on RL tasks.

The weighting mechanism accelerates adaptation to environment changes.

The approach is effective in diverse tasks like navigation and locomotion.

Abstract

Evolution strategies (ES), as a family of black-box optimization algorithms, recently emerge as a scalable alternative to reinforcement learning (RL) approaches such as Q-learning or policy gradient, and are much faster when many central processing units (CPUs) are available due to better parallelization. In this paper, we propose a systematic incremental learning method for ES in dynamic environments. The goal is to adjust previously learned policy to a new one incrementally whenever the environment changes. We incorporate an instance weighting mechanism with ES to facilitate its learning adaptation, while retaining scalability of ES. During parameter updating, higher weights are assigned to instances that contain more new knowledge, thus encouraging the search distribution to move towards new promising areas of parameter space. We propose two easy-to-implement metrics to calculate the weights: instance novelty and instance quality. Instance novelty measures an instance's difference from the previous optimum in the original environment, while instance quality corresponds to how well an instance performs in the new environment. The resulting algorithm, Instance Weighted Incremental Evolution Strategies (IW-IES), is verified to achieve significantly improved performance on challenging RL tasks ranging from robot navigation to locomotion. This paper thus introduces a family of scalable ES algorithms for RL domains that enables rapid learning adaptation to dynamic environments.