Neural 360$^\circ$ Structured Light with Learned Metasurfaces

TL;DR

This paper introduces a neural framework for designing learned metasurfaces that generate 360-degree structured light, significantly improving performance in holography and 3D imaging tasks through differentiable optimization and fast propagation modeling.

Contribution

It presents a novel differentiable framework for optimizing metasurface designs for 360-degree structured light, enabling complex pattern projection and enhanced 3D imaging accuracy.

Findings

Achieved first 360-degree complex pattern projection.

Propagation model is 50,000 times faster than traditional methods.

Improved depth-estimation RMSE by 5.09 times.

Abstract

Structured light has proven instrumental in 3D imaging, LiDAR, and holographic light projection. Metasurfaces, comprised of sub-wavelength-sized nanostructures, facilitate 180$^\circ$ field-of-view (FoV) structured light, circumventing the restricted FoV inherent in traditional optics like diffractive optical elements. However, extant metasurface-facilitated structured light exhibits sub-optimal performance in downstream tasks, due to heuristic pattern designs such as periodic dots that do not consider the objectives of the end application. In this paper, we present neural 360$^\circ$ structured light, driven by learned metasurfaces. We propose a differentiable framework, that encompasses a computationally-efficient 180$^\circ$ wave propagation model and a task-specific reconstructor, and exploits both transmission and reflection channels of the metasurface. Leveraging a first-order optimizer within our differentiable framework, we optimize the metasurface design, thereby realizing neural 360$^\circ$ structured light. We have utilized neural 360$^\circ$ structured light for holographic light projection and 3D imaging. Specifically, we demonstrate the first 360$^\circ$ light projection of complex patterns, enabled by our propagation model that can be computationally evaluated 50,000$\times$ faster than the Rayleigh-Sommerfeld propagation. For 3D imaging, we improve depth-estimation accuracy by 5.09$\times$ in RMSE compared to the heuristically-designed structured light. Neural 360$^\circ$ structured light promises robust 360$^\circ$ imaging and display for robotics, extended-reality systems, and human-computer interactions.