Reservoir Computing as a Language Model

TL;DR

This paper compares reservoir computing and transformer models for character-level language modeling, highlighting trade-offs in prediction accuracy and computational efficiency, and exploring attention-enhanced reservoirs for improved performance.

Contribution

It introduces a comparative analysis of reservoir computing approaches against transformers for language modeling, including novel attention-enhanced reservoirs, emphasizing efficiency and scalability.

Findings

Transformers outperform reservoirs in prediction quality.

Reservoir computing offers higher efficiency with faster training and inference.

Attention-enhanced reservoirs adapt dynamically, improving performance.

Abstract

Large Language Models (LLM) have dominated the science and media landscape duo to their impressive performance on processing large chunks of data and produce human-like levels of text. Nevertheless, their huge energy demand and slow processing are still a bottleneck to further increasing quality while also making the models accessible to everyone. To solve this bottleneck, we will investigate how reservoir computing performs on natural text processing, which could enable fast and energy efficient hardware implementations. Studies investigating the use of reservoir computing as a language model remain sparse. In this paper, we compare three distinct approaches for character-level language modeling, two different \emph{reservoir computing} approaches, where only an output layer is trainable, and the well-known \emph{transformer}-based architectures, which fully learn an attention-based sequence representation. We explore the performance, computational cost and prediction accuracy for both paradigms by equally varying the number of trainable parameters for all models. Using a consistent pipeline for all three approaches, we demonstrate that transformers excel in prediction quality, whereas reservoir computers remain highly efficient reducing the training and inference speed. Furthermore, we investigate two types of reservoir computing: a \emph{traditional reservoir} with a static linear readout, and an \emph{attention-enhanced reservoir} that dynamically adapts its output weights via an attention mechanism. Our findings underline how these paradigms scale and offer guidelines to balance resource constraints with performance.