Neurosim: A Fast Simulator for Neuromorphic Robot Perception

TL;DR

Neurosim is a high-performance, real-time simulation library for neuromorphic perception sensors and vehicle dynamics, enabling efficient training and testing of neuromorphic algorithms in robotics.

Contribution

It introduces Neurosim and Cortex, novel tools that significantly improve simulation speed and integration for neuromorphic robotics research.

Findings

Achieves up to ~2700 FPS on desktop GPU

Facilitates training of neuromorphic perception algorithms

Enables real-time testing in closed-loop systems

Abstract

Neurosim is a fast, real-time, high-performance library for simulating sensors such as dynamic vision sensors, RGB cameras, depth sensors, and inertial sensors. It can also simulate agile dynamics of multi-rotor vehicles in complex and dynamic environments. Neurosim can achieve frame rates as high as ~2700 FPS on a desktop GPU. Neurosim integrates with a ZeroMQ-based communication library called Cortex to facilitate seamless integration with machine learning and robotics workflows. Cortex provides a high-throughput, low-latency message-passing system for Python and C++ applications, with native support for NumPy arrays and PyTorch tensors. This paper discusses the design philosophy behind Neurosim and Cortex. It demonstrates how they can be used to (i) train neuromorphic perception and control algorithms, e.g., using self-supervised learning on time-synchronized multi-modal data, and (ii) test real-time implementations of these algorithms in closed-loop. Neurosim and Cortex are available at https://github.com/grasp-lyrl/neurosim .