Parallel bandit architecture based on laser chaos for reinforcement learning

TL;DR

This paper introduces a novel parallel bandit architecture for reinforcement learning that leverages photonic decision-making and laser chaos, demonstrating faster adaptation and unique state properties compared to traditional Q-learning.

Contribution

The study proposes the PBRL architecture, enabling multi-state reinforcement learning with photonic decision-makers, and shows its advantages over Q-learning in speed and environment adaptation.

Findings

PBRL adapts faster than Q-learning in a cart-pole task.

Chaotic laser sequences improve learning speed due to autocorrelation.

PBRL exhibits distinct state transition properties compared to Q-learning.

Abstract

Accelerating artificial intelligence by photonics is an active field of study aiming to exploit the unique properties of photons. Reinforcement learning is an important branch of machine learning, and photonic decision-making principles have been demonstrated with respect to the multi-armed bandit problems. However, reinforcement learning could involve a massive number of states, unlike previously demonstrated bandit problems where the number of states is only one. Q-learning is a well-known approach in reinforcement learning that can deal with many states. The architecture of Q-learning, however, does not fit well photonic implementations due to its separation of update rule and the action selection. In this study, we organize a new architecture for multi-state reinforcement learning as a parallel array of bandit problems in order to benefit from photonic decision-makers, which we call parallel bandit architecture for reinforcement learning or PBRL in short. Taking a cart-pole balancing problem as an instance, we demonstrate that PBRL adapts to the environment in fewer time steps than Q-learning. Furthermore, PBRL yields faster adaptation when operated with a chaotic laser time series than the case with uniformly distributed pseudorandom numbers where the autocorrelation inherent in the laser chaos provides a positive effect. We also find that the variety of states that the system undergoes during the learning phase exhibits completely different properties between PBRL and Q-learning. The insights obtained through the present study are also beneficial for existing computing platforms, not just photonic realizations, in accelerating performances by the PBRL algorithms and correlated random sequences.