Towards Signal Processing In Large Language Models

TL;DR

This paper explores integrating signal processing techniques into large language models, using learnable time-frequency representations to improve convergence and performance with minimal additional parameters.

Contribution

It introduces a novel approach of applying Fourier-like transforms within LLMs, enhancing training efficiency and accuracy.

Findings

Faster convergence in GPT-like models

Performance improvements with minimal extra parameters

Potential for new signal processing algorithms in neural networks

Abstract

This paper introduces the idea of applying signal processing inside a Large Language Model (LLM). With the recent explosion of generative AI, our work can help bridge two fields together, namely the field of signal processing and large language models. We draw parallels between classical Fourier-Transforms and Fourier Transform-like learnable time-frequency representations for every intermediate activation signal of an LLM. Once we decompose every activation signal across tokens into a time-frequency representation, we learn how to filter and reconstruct them, with all components learned from scratch, to predict the next token given the previous context. We show that for GPT-like architectures, our work achieves faster convergence and significantly increases performance by adding a minuscule number of extra parameters when trained for the same epochs. We hope this work paves the way for algorithms exploring signal processing inside the signals found in neural architectures like LLMs and beyond.