Efficient Reinforcement Learning On Passive RRAM Crossbar Array

TL;DR

This paper presents a novel Monte Carlo reinforcement learning implementation on passive RRAM crossbar arrays, achieving significant improvements in area efficiency and robustness against device variations and endurance issues.

Contribution

It introduces the first algorithm-hardware co-design for Monte Carlo RL on passive RRAM arrays, overcoming endurance limitations and outperforming prior digital and active RRAM implementations.

Findings

Outperforms prior implementations by over five orders of magnitude in area.

Demonstrates robustness against RRAM variations and endurance failures.

Achieves energy-efficient RL suitable for embedded neuromorphic systems.

Abstract

The unprecedented growth in the field of machine learning has led to the development of deep neuromorphic networks trained on labelled dataset with capability to mimic or even exceed human capabilities. However, for applications involving continuous decision making in unknown environments, such as rovers for space exploration, robots, unmanned aerial vehicles, etc., explicit supervision and generation of labelled data set is extremely difficult and expensive. Reinforcement learning (RL) allows the agents to take decisions without any (human/external) supervision or training on labelled dataset. However, the conventional implementations of RL on advanced digital CPUs/GPUs incur a significantly large power dissipation owing to their inherent von-Neumann architecture. Although crossbar arrays of emerging non-volatile memories such as resistive (R)RAMs with their innate capability to perform energy-efficient in situ multiply-accumulate operation appear promising for Q-learning-based RL implementations, their limited endurance restricts their application in practical RL systems with overwhelming weight updates. To address this issue and realize the true potential of RRAM-based RL implementations, in this work, for the first time, we perform an algorithm-hardware co-design and propose a novel implementation of Monte Carlo (MC) RL algorithm on passive RRAM crossbar array. We analyse the performance of the proposed MC RL implementation on the classical cart-pole problem and demonstrate that it not only outperforms the prior digital and active 1-Transistor-1-RRAM (1T1R)-based implementations by more than five orders of magnitude in terms of area but is also robust against the spatial and temporal variations and endurance failure of RRAMs.