LAS: Loss-less ANN-SNN Conversion for Fully Spike-Driven Large Language Models

TL;DR

LAS introduces a novel loss-less conversion method for fully spike-driven LLMs, effectively handling activation outliers and nonlinearities, resulting in improved accuracy and energy efficiency in spiking models.

Contribution

LAS proposes two new neurons and tailored transformer components to enable loss-less ANN-to-SNN conversion for large language models.

Findings

Achieves loss-less conversion on multiple models

Improves accuracy by 2% on OPT-66B WSC task

Verifies effectiveness through parameter and ablation studies

Abstract

Spiking Large Language Models (LLMs) have emerged as an energy-efficient alternative to conventional LLMs through their event-driven computation. To effectively obtain spiking LLMs, researchers develop different ANN-to-SNN conversion methods by leveraging pre-trained ANN parameters while inheriting the energy efficiency of SNN. However, existing conversion methods struggle with extreme activation outliers and incompatible nonlinear operations of ANN-based LLMs. To address this, we propose a loss-less ANN-SNN conversion for fully spike-driven LLMs, termed LAS. Specifically, LAS introduces two novel neurons to convert the activation outlier and nonlinear operation of ANN-based LLMs. Moreover, LAS tailors the spike-equivalent Transformer components for spiking LLMs, which can ensure full spiking conversion without any loss of performance. Experimental results on six language models and two vision-language models demonstrate that LAS achieves loss-less conversion. Notably, on OPT-66B, LAS even improves the accuracy of 2\% on the WSC task. In addition, the parameter and ablation studies further verify the effectiveness of LAS. The source code is available at https://github.com/lc783/LAS