Fusing Event-based Camera and Radar for SLAM Using Spiking Neural Networks with Continual STDP Learning

TL;DR

This paper introduces a novel SLAM system that fuses event-based camera and radar data processed by bio-inspired spiking neural networks with continual STDP learning, enabling real-time, offline-training-free drone navigation.

Contribution

It presents the first SLAM architecture combining event camera and radar with SNNs using STDP, eliminating the need for offline training and improving robustness in challenging lighting conditions.

Findings

Outperforms state-of-the-art RGB-based SLAM methods in robustness.

Operates without offline training, learning features online.

Demonstrates effective loop closure detection and map correction.

Abstract

This work proposes a first-of-its-kind SLAM architecture fusing an event-based camera and a Frequency Modulated Continuous Wave (FMCW) radar for drone navigation. Each sensor is processed by a bio-inspired Spiking Neural Network (SNN) with continual Spike-Timing-Dependent Plasticity (STDP) learning, as observed in the brain. In contrast to most learning-based SLAM systems%, which a) require the acquisition of a representative dataset of the environment in which navigation must be performed and b) require an off-line training phase, our method does not require any offline training phase, but rather the SNN continuously learns features from the input data on the fly via STDP. At the same time, the SNN outputs are used as feature descriptors for loop closure detection and map correction. We conduct numerous experiments to benchmark our system against state-of-the-art RGB methods and we demonstrate the robustness of our DVS-Radar SLAM approach under strong lighting variations.