Improving Performance of Spike-based Deep Q-Learning using Ternary Neurons

TL;DR

This paper introduces a novel ternary spiking neuron model that enhances deep Q-learning performance by reducing gradient bias, outperforming binary neurons in Atari game benchmarks for autonomous decision-making.

Contribution

A new ternary spiking neuron model is proposed to address gradient bias, improving deep Q-learning performance over existing binary neuron models.

Findings

Ternary neurons mitigate performance degradation in Q-learning.

Proposed model outperforms binary neurons in Atari benchmarks.

Reduces gradient estimation bias in training.

Abstract

We propose a new ternary spiking neuron model to improve the representation capacity of binary spiking neurons in deep Q-learning. Although a ternary neuron model has recently been introduced to overcome the limited representation capacity offered by the binary spiking neurons, we show that its performance is worse than that of binary models in deep Q-learning tasks. We hypothesize gradient estimation bias during the training process as the underlying potential cause through mathematical and empirical analysis. We propose a novel ternary spiking neuron model to mitigate this issue by reducing the estimation bias. We use the proposed ternary spiking neuron as the fundamental computing unit in a deep spiking Q-learning network (DSQN) and evaluate the network's performance in seven Atari games from the Gym environment. Results show that the proposed ternary spiking neuron mitigates the drastic performance degradation of ternary neurons in Q-learning tasks and improves the network performance compared to the existing binary neurons, making DSQN a more practical solution for on-board autonomous decision-making tasks.