Resistive memory-based zero-shot liquid state machine for multimodal event data learning

TL;DR

This paper introduces a resistive memory-based liquid state machine integrated with trainable neural projections for zero-shot learning of multimodal event data, achieving high efficiency and accuracy on neuromorphic hardware.

Contribution

It presents a novel hardware-software co-design of a liquid state machine with in-memory computing for zero-shot multimodal learning, reducing training costs and energy consumption.

Findings

Achieves comparable accuracy to software models on multimodal datasets.

Reduces training costs by over 150-fold compared to state-of-the-art methods.

Improves energy efficiency by over 23-fold on neuromorphic hardware.

Abstract

The human brain is a complex spiking neural network (SNN), capable of learning multimodal signals in a zero-shot manner by generalizing existing knowledge. Remarkably, it maintains minimal power consumption through event-based signal propagation. However, replicating the human brain in neuromorphic hardware presents both hardware and software challenges. Hardware limitations, such as the slowdown of Moore's law and Von Neumann bottleneck, hinder the efficiency of digital computers. Additionally, SNNs are characterized by their software training complexities. To this end, we propose a hardware-software co-design on a 40 nm 256 Kb in-memory computing macro that physically integrates a fixed and random liquid state machine (LSM) SNN encoder with trainable artificial neural network (ANN) projections. We showcase the zero-shot LSM-based learning of multimodal events on the N-MNIST and N-TIDIGITS datasets, including visual and audio data association, as well as neural and visual data alignment for brain-machine interfaces. Our co-design achieves classification accuracy comparable to fully optimized software models, resulting in a 152.83 and 393.07-fold reduction in training costs compared to SOTA contrastive language-image pre-training (CLIP) and Prototypical networks, and a 23.34 and 160-fold improvement in energy efficiency compared to cutting-edge digital hardware, respectively. These proof-of-principle prototypes demonstrate zero-shot multimodal events learning capability for emerging efficient and compact neuromorphic hardware.