Direct Kernel Optimization: Efficient Design for Opto-Electronic Convolutional Neural Networks

TL;DR

This paper introduces Direct Kernel Optimization (DKO), a two-stage training method for hybrid opto-electronic neural networks that improves efficiency and accuracy by synthesizing optical kernels after initial electronic training.

Contribution

DKO reduces the computational complexity of joint optical-electronic training by separating the optimization into two stages, enabling scalable and effective hybrid neural network training.

Findings

DKO achieves twice the accuracy of end-to-end training under similar computational budgets.

DKO reduces training time compared to traditional joint optimization methods.

Simulation results demonstrate DKO's effectiveness for monocular depth estimation.

Abstract

Hybrid opto-electronic neural networks combine optical front-ends with electronic back-ends to perform vision tasks, but joint end-to-end (E2E) optimization of optical and electronic components is computationally expensive due to large parameter spaces and repeated optical convolutions. We propose Direct Kernel Optimization (DKO), a two-stage training framework that first trains a conventional electronic CNN and then synthesizes optical kernels to replicate the first-layer convolutional filters, reducing optimization dimensionality and avoiding hefty simulated optical convolutions during optimization. We evaluate DKO in simulation on a monocular depth estimation model and show that it achieves twice the accuracy of E2E training under equal computational budgets while reducing training time. Given the substantial computational challenges of optimizing hybrid opto-electronic systems, our results position DKO as a scalable optimization approach to train and realize these systems.