Hard-Thresholding Meets Evolution Strategies in Reinforcement Learning

TL;DR

This paper introduces NESHT, a novel method combining Hard-Thresholding with Natural Evolution Strategies to improve feature relevance in reinforcement learning, especially in noisy, real-world scenarios.

Contribution

It proposes NESHT, a new approach that enhances NES by promoting sparsity, effectively filtering out irrelevant features in reinforcement learning tasks.

Findings

NESHT outperforms standard NES in noisy Mujoco and Atari tasks.

It effectively mitigates the impact of irrelevant features.

Empirical results show improved decision-making performance.

Abstract

Evolution Strategies (ES) have emerged as a competitive alternative for model-free reinforcement learning, showcasing exemplary performance in tasks like Mujoco and Atari. Notably, they shine in scenarios with imperfect reward functions, making them invaluable for real-world applications where dense reward signals may be elusive. Yet, an inherent assumption in ES, that all input features are task-relevant, poses challenges, especially when confronted with irrelevant features common in real-world problems. This work scrutinizes this limitation, particularly focusing on the Natural Evolution Strategies (NES) variant. We propose NESHT, a novel approach that integrates Hard-Thresholding (HT) with NES to champion sparsity, ensuring only pertinent features are employed. Backed by rigorous analysis and empirical tests, NESHT demonstrates its promise in mitigating the pitfalls of irrelevant features and shines in complex decision-making problems like noisy Mujoco and Atari tasks.