Model selection in hybrid quantum neural networks with applications to quantum transformer architectures

TL;DR

This paper introduces the Quantum Bias-Expressivity Toolbox ($ exttt{QBET}$), a framework for evaluating hybrid quantum-classical transformer models, enabling efficient model selection and demonstrating quantum advantages in specific tasks.

Contribution

The paper develops $ exttt{QBET}$ with novel metrics for bias and expressivity, facilitating pre-screening of quantum models without full training, and applies it to quantum transformer architectures.

Findings

Quantum self-attention can outperform classical models in certain scenarios.

$ exttt{QBET}$ effectively predicts promising quantum model variants.

Efficient evaluation reduces resource-intensive training processes.

Abstract

Quantum machine learning models generally lack principled design guidelines, often requiring full resource-intensive training across numerous choices of encodings, quantum circuit designs and initialization strategies to find effective configuration. To address this challenge, we develope the Quantum Bias-Expressivity Toolbox ($\texttt{QBET}$), a framework for evaluating quantum, classical, and hybrid transformer architectures. In this toolbox, we introduce lean metrics for Simplicity Bias ($\texttt{SB}$) and Expressivity ($\texttt{EXP}$), for comparing across various models, and extend the analysis of $\texttt{SB}$ to generative and multiclass-classification tasks. We show that $\texttt{QBET}$ enables efficient pre-screening of promising model variants obviating the need to execute complete training pipelines. In evaluations on transformer-based classification and generative tasks we employ a total of $18$ qubits for embeddings ($6$ qubits each for query, key, and value). We identify scenarios in which quantum self-attention variants surpass their classical counterparts by ranking the respective models according to the $\texttt{SB}$ metric and comparing their relative performance.