The More the Merrier: Running Multiple Neuromorphic Components On-Chip for Robotic Control

TL;DR

This paper demonstrates a novel on-chip pipeline for vision-based robotic control that runs multiple neuromorphic networks concurrently on hardware, enabling complex tasks with low power and latency.

Contribution

It introduces a first-of-its-kind on-chip pipeline using a spiking neural state machine for orchestrating multiple neuromorphic networks in robotic control.

Findings

All components run concurrently on-chip in the milli Watt regime.

Achieves low-latency performance comparable to state-of-the-art.

Successfully tested on real robotic arm for a complex task.

Abstract

It has long been realized that neuromorphic hardware offers benefits for the domain of robotics such as low energy, low latency, as well as unique methods of learning. In aiming for more complex tasks, especially those incorporating multimodal data, one hurdle continuing to prevent their realization is an inability to orchestrate multiple networks on neuromorphic hardware without resorting to off-chip process management logic. To address this, we show a first example of a pipeline for vision-based robot control in which numerous complex networks can be run entirely on hardware via the use of a spiking neural state machine for process orchestration. The pipeline is validated on the Intel Loihi 2 research chip. We show that all components can run concurrently on-chip in the milli Watt regime at latencies competitive with the state-of-the-art. An equivalent network on simulated hardware is shown to accomplish robotic arm plug insertion in simulation, and the core elements of the pipeline are additionally tested on a real robotic arm.