Spikes as regularizers

TL;DR

The paper introduces SPIRAL, a novel confidence-based learning algorithm that uses spike encoding to improve robustness and reduce overfitting in single-layer neural networks.

Contribution

It presents a new spike-regularized learning method, SPIRAL, which adaptively updates weights based on confidence estimates and activation offsets, inspired by neurophysiological observations.

Findings

SPIRAL outperforms averaged perceptron and AROW in robustness.

SPIRAL is less prone to overfitting.

The algorithm demonstrates improved stability in experiments.

Abstract

We present a confidence-based single-layer feed-forward learning algorithm SPIRAL (Spike Regularized Adaptive Learning) relying on an encoding of activation spikes. We adaptively update a weight vector relying on confidence estimates and activation offsets relative to previous activity. We regularize updates proportionally to item-level confidence and weight-specific support, loosely inspired by the observation from neurophysiology that high spike rates are sometimes accompanied by low temporal precision. Our experiments suggest that the new learning algorithm SPIRAL is more robust and less prone to overfitting than both the averaged perceptron and AROW.