Reservoir Computing inspired Matrix Multiplication-free Language Model

TL;DR

This paper introduces a matrix multiplication-free language model inspired by reservoir computing, significantly reducing computational costs and training time while maintaining performance.

Contribution

It proposes a novel architecture that combines reservoir layers with fixed and shared weights to improve efficiency without extra training overhead.

Findings

Parameters reduced by up to 19%

Training time decreased by 9.9%

Inference time decreased by 8.0%

Abstract

Large language models (LLMs) have achieved state-of-the-art performance in natural language processing; however, their high computational cost remains a major bottleneck. In this study, we target computational efficiency by focusing on a matrix multiplication free language model (MatMul-free LM) and further reducing the training cost through an architecture inspired by reservoir computing. Specifically, we partially fix and share the weights of selected layers in the MatMul-free LM and insert reservoir layers to obtain rich dynamic representations without additional training overhead. Additionally, several operations are combined to reduce memory accesses. Experimental results show that the proposed architecture reduces the number of parameters by up to 19%, training time by 9.9%, and inference time by 8.0%, while maintaining comparable performance to the baseline model.