Lightening-Transformer: A Dynamically-operated Optically-interconnected Photonic Transformer Accelerator
Hanqing Zhu, Jiaqi Gu, Hanrui Wang, Zixuan Jiang, Zhekai Zhang,, Rongxing Tang, Chenghao Feng, Song Han, Ray T. Chen, David Z. Pan

TL;DR
This paper introduces Lightening-Transformer, a novel photonic accelerator with dynamically-operated cores that significantly improves energy efficiency and latency for Transformer models, outperforming prior photonic and electronic accelerators.
Contribution
It presents the first dynamically-operated photonic tensor core and an integrated accelerator design tailored for Transformer workloads, enabling high performance and energy efficiency.
Findings
Achieves >2.6x energy reduction compared to prior photonic accelerators.
Achieves >12x latency reduction compared to prior photonic accelerators.
Delivers 2-3 orders of magnitude lower energy-delay product than electronic Transformer accelerators.
Abstract
The wide adoption and significant computing resource of attention-based transformers, e.g., Vision Transformers and large language models (LLM), have driven the demand for efficient hardware accelerators. There is a growing interest in exploring photonics as an alternative technology to digital electronics due to its high energy efficiency and ultra-fast processing speed. Photonic accelerators have shown promising results for CNNs, which mainly rely on weight-static linear operations. However, they encounter issues when efficiently supporting Transformer architectures, questioning the applicability of photonics to advanced ML tasks. The primary hurdle lies in their inefficiency in handling unique workloads in Transformers, i.e., dynamic and full-range tensor multiplication. In this work, we propose Lightening-Transformer, the first light-empowered, high-performance, and energy-efficient…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNeural Networks and Reservoir Computing · Photonic and Optical Devices · Optical Network Technologies
