Explore Activation Sparsity in Recurrent LLMs for Energy-Efficient Neuromorphic Computing

TL;DR

This paper introduces a training-free method to sparsify activations in recurrent large language models, significantly reducing energy consumption and latency on neuromorphic hardware while maintaining accuracy.

Contribution

It presents a novel, low-cost activation sparsification algorithm for R-LLMs that enhances energy efficiency without additional training, applicable to various LLM architectures.

Findings

Significant reduction in computational demands.

Notable energy savings and latency improvements on neuromorphic hardware.

Maintains competitive accuracy across benchmarks.

Abstract

The recent rise of Large Language Models (LLMs) has revolutionized the deep learning field. However, the desire to deploy LLMs on edge devices introduces energy efficiency and latency challenges. Recurrent LLM (R-LLM) architectures have proven effective in mitigating the quadratic complexity of self-attention, making them a potential paradigm for computing on-edge neuromorphic processors. In this work, we propose a low-cost, training-free algorithm to sparsify R-LLMs' activations to enhance energy efficiency on neuromorphic hardware. Our approach capitalizes on the inherent structure of these models, rendering them well-suited for energy-constrained environments. Although primarily designed for R-LLMs, this method can be generalized to other LLM architectures, such as transformers, as demonstrated on the OPT model, achieving comparable sparsity and efficiency improvements. Empirical studies illustrate that our method significantly reduces computational demands while maintaining competitive accuracy across multiple zero-shot learning benchmarks. Additionally, hardware simulations with the SENECA neuromorphic processor underscore notable energy savings and latency improvements. These results pave the way for low-power, real-time neuromorphic deployment of LLMs and demonstrate the feasibility of training-free on-chip adaptation using activation sparsity.