Instantaneous Stereo Depth Estimation of Real-World Stimuli with a Neuromorphic Stereo-Vision Setup

TL;DR

This paper demonstrates a neuromorphic stereo-vision system that uses event cameras and spiking neural networks to estimate depth in real-time with real-world stimuli, inspired by biological stereo-matching.

Contribution

It validates a brain-inspired, event-based stereo-matching architecture on a neuromorphic processor using real-world data, enabling instant depth estimation.

Findings

Provides coarse disparity estimates instantaneously

Detects moving stimuli in depth in real-time

Validates on real-world dataset with neuromorphic hardware

Abstract

The stereo-matching problem, i.e., matching corresponding features in two different views to reconstruct depth, is efficiently solved in biology. Yet, it remains the computational bottleneck for classical machine vision approaches. By exploiting the properties of event cameras, recently proposed Spiking Neural Network (SNN) architectures for stereo vision have the potential of simplifying the stereo-matching problem. Several solutions that combine event cameras with spike-based neuromorphic processors already exist. However, they are either simulated on digital hardware or tested on simplified stimuli. In this work, we use the Dynamic Vision Sensor 3D Human Pose Dataset (DHP19) to validate a brain-inspired event-based stereo-matching architecture implemented on a mixed-signal neuromorphic processor with real-world data. Our experiments show that this SNN architecture, composed of coincidence detectors and disparity sensitive neurons, is able to provide a coarse estimate of the input disparity instantaneously, thereby detecting the presence of a stimulus moving in depth in real-time.