# Multimodal deep learning using on-chip diffractive optics with in situ training capability

## TL;DR

This paper introduces a new photonic chip that can perform multimodal deep learning with high efficiency and accuracy by using on-chip diffractive optics.

## Contribution

The novel contribution is a trainable diffractive optical neural network chip (TDONN) that enables multimodal deep learning with in situ training and high performance.

## Key findings

- The TDONN chip achieves 217.6 TOPS throughput with high computing density and energy efficiency.
- It successfully performs four-class classification across vision, audio, and touch modalities with 85.7% accuracy.
- The chip enables in situ training and fast convergence using a customized stochastic gradient descent algorithm.

## Abstract

Multimodal deep learning plays a pivotal role in supporting the processing and learning of diverse data types within the realm of artificial intelligence generated content (AIGC). However, most photonic neuromorphic processors for deep learning can only handle a single data modality (either vision or audio) due to the lack of abundant parameter training in optical domain. Here, we propose and demonstrate a trainable diffractive optical neural network (TDONN) chip based on on-chip diffractive optics with massive tunable elements to address these constraints. The TDONN chip includes one input layer, five hidden layers, and one output layer, and only one forward propagation is required to obtain the inference results without frequent optical-electrical conversion. The customized stochastic gradient descent algorithm and the drop-out mechanism are developed for photonic neurons to realize in situ training and fast convergence in the optical domain. The TDONN chip achieves a potential throughput of 217.6 tera-operations per second (TOPS) with high computing density (447.7 TOPS/mm2), high system-level energy efficiency (7.28 TOPS/W), and low optical latency (30.2 ps). The TDONN chip has successfully implemented four-class classification in different modalities (vision, audio, and touch) and achieve 85.7% accuracy on multimodal test sets. Our work opens up a new avenue for multimodal deep learning with integrated photonic processors, providing a potential solution for low-power AI large models using photonic technology.

Most photonic processors can only handle a single data modality due to the lack of abundant parameter training in optical domain. Here, authors propose and demonstrate a trainable diffractive optical neural network chip based on on-chip diffractive optics with tunable elements to address these constraints.