PGKET: A Photonic Gaussian Kernel Enhanced Transformer

TL;DR

The paper introduces PGKET, a novel photonic Gaussian kernel enhanced transformer that leverages photon interferometry for efficient self-attention, demonstrating superior performance on image classification benchmarks and promising faster convergence in photonic computing.

Contribution

It proposes PGKET with PGKSAM, utilizing photonic Gaussian kernels for parallel self-attention computation, advancing efficiency and performance in transformer models.

Findings

Outperforms state-of-the-art transformers on MedMNIST v2 and CIFAR-10.

Uses photon interferometry for parallel attention score calculation.

Accelerates convergence in photonic computing and machine learning.

Abstract

Self-Attention Mechanisms (SAMs) enhance model performance by extracting key information but are inefficient when dealing with long sequences. To this end, a photonic Gaussian Kernel Enhanced Transformer (PGKET) is proposed, based on the Photonic Gaussian Kernel Self-Attention Mechanism (PGKSAM). The PGKSAM calculates the Photonic Gaussian Kernel Self-Attention Score (PGKSAS) using photon interferometry and superposition to process multiple inputs in parallel. Experimental results show that PGKET outperforms some state-of-the-art transformers in multi-classification tasks on MedMNIST v2 and CIFAR-10, and is expected to improve performance in complex tasks and accelerate the convergence of Photonic Computing (PC) and machine learning.