Efficient Off-Policy Reinforcement Learning via Brain-Inspired Computing

TL;DR

This paper introduces QHD, a brain-inspired, hyperdimensional reinforcement learning method that significantly improves efficiency and real-time learning capabilities over traditional deep neural network-based RL algorithms like DQN.

Contribution

QHD is a novel, lightweight, brain-inspired RL model that achieves higher efficiency and comparable or better rewards than DQN, suitable for real-time and embedded applications.

Findings

QHD achieves 12.3x speedup with minimal quality loss.

QHD provides 34.6x speedup and better learning quality than DQN.

QHD is effective on both desktop and embedded platforms.

Abstract

Reinforcement Learning (RL) has opened up new opportunities to enhance existing smart systems that generally include a complex decision-making process. However, modern RL algorithms, e.g., Deep Q-Networks (DQN), are based on deep neural networks, resulting in high computational costs. In this paper, we propose QHD, an off-policy value-based Hyperdimensional Reinforcement Learning, that mimics brain properties toward robust and real-time learning. QHD relies on a lightweight brain-inspired model to learn an optimal policy in an unknown environment. On both desktop and power-limited embedded platforms, QHD achieves significantly better overall efficiency than DQN while providing higher or comparable rewards. QHD is also suitable for highly-efficient reinforcement learning with great potential for online and real-time learning. Our solution supports a small experience replay batch size that provides 12.3 times speedup compared to DQN while ensuring minimal quality loss. Our evaluation shows QHD capability for real-time learning, providing 34.6 times speedup and significantly better quality of learning than DQN.