Neural Episodic Control with State Abstraction

TL;DR

NECSA enhances deep reinforcement learning by integrating state abstraction into episodic control, leading to improved sample efficiency and better utilization of historical experience in complex tasks.

Contribution

This paper introduces NECSA, a novel state abstraction-based episodic control method that improves scalability and leverages latent information for more efficient learning.

Findings

NECSA outperforms existing episodic control methods on MuJoCo and Atari tasks.

NECSA achieves higher sample efficiency in complex reinforcement learning environments.

The approach effectively utilizes historical behaviors through state abstraction.

Abstract

Existing Deep Reinforcement Learning (DRL) algorithms suffer from sample inefficiency. Generally, episodic control-based approaches are solutions that leverage highly-rewarded past experiences to improve sample efficiency of DRL algorithms. However, previous episodic control-based approaches fail to utilize the latent information from the historical behaviors (e.g., state transitions, topological similarities, etc.) and lack scalability during DRL training. This work introduces Neural Episodic Control with State Abstraction (NECSA), a simple but effective state abstraction-based episodic control containing a more comprehensive episodic memory, a novel state evaluation, and a multi-step state analysis. We evaluate our approach to the MuJoCo and Atari tasks in OpenAI gym domains. The experimental results indicate that NECSA achieves higher sample efficiency than the state-of-the-art episodic control-based approaches. Our data and code are available at the project website\footnote{\url{https://sites.google.com/view/drl-necsa}}.