Quantum Attention by Overlap Interference: Predicting Sequences from Classical and Many-Body Quantum Data

TL;DR

This paper introduces a quantum self-attention mechanism that leverages interference of state overlaps for sequence prediction, demonstrating potential advantages over classical methods in quantum and classical data modeling.

Contribution

It presents a variational quantum implementation of self-attention that uses interference for nonlinearity and directly computes loss, offering a new approach for quantum sequence modeling.

Findings

QSA scales as O(T d^2), potentially outperforming classical O(T^2 d) in certain regimes.

QSA successfully learns sequence prediction on classical data.

QSA models quantum trajectories, showing versatility in data types.

Abstract

We propose a variational quantum implementation of self-attention (QSA), the core operation in transformers and large language models, which predicts future elements of a sequence by forming overlap-weighted combinations of past data. At variance with previous approaches, our QSA realizes the required nonlinearity through interference of state overlaps and returns a Renyi-1/2 cross-entropy loss directly as the expectation value of an observable, avoiding the need to decode amplitude-encoded predictions into classical logits. Furthermore, QSA naturally accommodates a constrained, trainable data-embedding that ties quantum state overlaps to data-level similarities. We find a gate complexity dominant scaling O(T d^2) for QSA, versus O(T^2 d) classically, suggesting an advantage in the practical regime where the sequence length T dominates the embedding size d. In simulations, we show that our QSA-based quantum transformer learns sequence prediction on classical data and on many-body transverse-field Ising quantum trajectories, establishing trainable attention as a practical primitive for quantum dynamical modeling.