A bio-inspired minimal model for non-stationary K-armed bandits

TL;DR

This paper introduces a biologically inspired neural model for solving multi-armed bandit problems that performs as well as standard algorithms.

Contribution

A novel bio-inspired neural model that matches standard algorithms and adapts behavior based on uncertainty.

Findings

The model performs robustly across stochastic bandit problems, matching Thompson Sampling and UCB.

It adapts strategies, being greedy in low-uncertainty and exploratory in high-uncertainty scenarios.

Hyperparameters evolved to values consistent with synaptic mechanisms and learning rate adaptation.

Abstract

While reinforcement learning algorithms have made significant progress in solving multi-armed bandit problems, they often lack biological plausibility in architecture and dynamics. Here, we propose a bio-inspired neural model based on interacting populations of rate neurons, drawing inspiration from the orbitofrontal cortex and anterior cingulate cortex. Our model reports robust performance across various stochastic bandit problems, matching the effectiveness of standard algorithms such as Thompson Sampling and UCB. Notably, the model exhibits adaptive behavior: employing greedy strategies in low-uncertainty situations while increasing exploratory behavior as uncertainty rises. Through evolutionary optimization, the model’s hyperparameters converged to values that align with the principles of synaptic mechanisms, particularly in terms of synapse-dependent neural activity and learning rate adaptation. These findings suggest that biologically-inspired computational architectures can achieve competitive performance while providing insights into neural mechanisms of decision-making under uncertainty.

Figures