Quantum-Inspired Self-Attention in a Large Language Model

TL;DR

This paper introduces a quantum-inspired self-attention mechanism integrated into GPT-1, demonstrating improved language modeling performance with only a modest increase in inference time.

Contribution

The paper presents the first integration of quantum-inspired self-attention into a large language model, enhancing performance over standard self-attention in language tasks.

Findings

15.5x better character error rate

4.7x better word error rate

13x lower cross-entropy loss

Abstract

Recent advances in Natural Language Processing have been predominantly driven by transformer-based architectures, which rely heavily on self-attention mechanisms to model relationships between tokens in a sequence. Similarly, the field of Quantum Natural Language Processing, which seeks to leverage quantum principles to address challenges in language understanding and generation tasks, has seen the recent development of quantum self-attention mechanisms. We propose a classical quantum-inspired self-attention (QISA) mechanism and integrate it into the full autoregressive language modeling pipeline of GPT-1. To the best of our knowledge, this is the first integration of this kind, as previous quantum self-attention mechanisms have been primarily tested on text classification. In our experiments, QISA achieves better performance when compared to standard self-attention on the metrics character error rate ($15.5\times$ better), word error rate ($4.7 \times $) and cross-entropy loss ($13 \times$). This is achieved while only requiring a $ 2.6\times$ longer inference time.