Sorbet: A Neuromorphic Hardware-Compatible Transformer-Based Spiking Language Model

TL;DR

Sorbet is a neuromorphic hardware-compatible transformer-based spiking language model that uses novel energy-efficient operations, achieving significant energy savings while maintaining competitive performance on language understanding benchmarks.

Contribution

Introduces Sorbet, a transformer-based spiking language model with novel softmax and normalization techniques optimized for neuromorphic hardware.

Findings

Achieves 27.16× energy savings over BERT.

Maintains competitive performance on GLUE benchmark.

Uses knowledge distillation and quantization for model compression.

Abstract

For reasons such as privacy, there are use cases for language models at the edge. This has given rise to small language models targeted for deployment in resource-constrained devices where energy efficiency is critical. Spiking neural networks (SNNs) offer a promising solution due to their energy efficiency, and there are already works on realizing transformer-based models on SNNs. However, key operations like softmax and layer normalization (LN) are difficult to implement on neuromorphic hardware, and many of these early works sidestepped them. To address these challenges, we introduce Sorbet, a transformer-based spiking language model that is more neuromorphic hardware-compatible. Sorbet incorporates a novel shifting-based softmax called PTsoftmax and a Bit Shifting PowerNorm (BSPN), both designed to replace the respective energy-intensive operations. By leveraging knowledge distillation and model quantization, Sorbet achieved a highly compressed binary weight model that maintains competitive performance while achieving $27.16\times$ energy savings compared to BERT. We validate Sorbet through extensive testing on the GLUE benchmark and a series of ablation studies, demonstrating its potential as an energy-efficient solution for language model inference. Our code is publicly available at \href{https://github.com/Kaiwen-Tang/Sorbet}{https://github.com/Kaiwen-Tang/Sorbet}