Tensorized Optical Multimodal Fusion Network

TL;DR

This paper introduces a tensorized optical multimodal fusion network that leverages self-attention and low-rank tensor fusion to significantly reduce hardware needs and boost energy efficiency in multimodal processing.

Contribution

It presents the first tensorized optical multimodal fusion architecture combining self-attention and low-rank tensor fusion for improved efficiency.

Findings

51.3 times reduction in hardware requirements

3.7×10^{13} MAC/J energy efficiency

First tensorized optical multimodal fusion network

Abstract

We propose the first tensorized optical multimodal fusion network architecture with a self-attention mechanism and low-rank tensor fusion. Simulation results show $51.3 \times$ less hardware requirement and $3.7\times 10^{13}$ MAC/J energy efficiency.