Unsupervised Learning of a Hierarchical Spiking Neural Network for Optical Flow Estimation: From Events to Global Motion Perception

TL;DR

This paper introduces a novel hierarchical spiking neural network architecture that learns optical flow and motion perception in an unsupervised manner from event-based camera data, mimicking biological visual motion systems.

Contribution

It presents the first hierarchical spiking neural network for optical flow estimation that learns motion selectivity without supervision, using a new neuron model and plasticity rule.

Findings

Emerges local and global motion selectivity in the network

Achieves accurate optical flow estimation on synthetic and real data

Provides a GPU-accelerated simulation framework for large-scale spiking networks

Abstract

The combination of spiking neural networks and event-based vision sensors holds the potential of highly efficient and high-bandwidth optical flow estimation. This paper presents the first hierarchical spiking architecture in which motion (direction and speed) selectivity emerges in an unsupervised fashion from the raw stimuli generated with an event-based camera. A novel adaptive neuron model and stable spike-timing-dependent plasticity formulation are at the core of this neural network governing its spike-based processing and learning, respectively. After convergence, the neural architecture exhibits the main properties of biological visual motion systems, namely feature extraction and local and global motion perception. Convolutional layers with input synapses characterized by single and multiple transmission delays are employed for feature and local motion perception, respectively; while global motion selectivity emerges in a final fully-connected layer. The proposed solution is validated using synthetic and real event sequences. Along with this paper, we provide the cuSNN library, a framework that enables GPU-accelerated simulations of large-scale spiking neural networks. Source code and samples are available at https://github.com/tudelft/cuSNN.