MetaScope: Optics-Driven Neural Network for Ultra-Micro Metalens Endoscopy

TL;DR

MetaScope introduces an optics-driven neural network designed specifically for ultra-micro metalens endoscopy, addressing unique optical challenges and improving imaging performance in biomedical applications.

Contribution

The paper presents a novel neural network architecture tailored for metalens endoscopy, incorporating optics-informed modules and a gradient-guided distillation for enhanced imaging.

Findings

MetaScope outperforms existing methods in metalens segmentation and restoration.

It effectively mitigates optical issues like intensity decay and chromatic aberration.

Demonstrates strong generalization in real biomedical scenes.

Abstract

Miniaturized endoscopy has advanced accurate visual perception within the human body. Prevailing research remains limited to conventional cameras employing convex lenses, where the physical constraints with millimetre-scale thickness impose serious impediments on the micro-level clinical. Recently, with the emergence of meta-optics, ultra-micro imaging based on metalenses (micron-scale) has garnered great attention, serving as a promising solution. However, due to the physical difference of metalens, there is a large gap in data acquisition and algorithm research. In light of this, we aim to bridge this unexplored gap, advancing the novel metalens endoscopy. First, we establish datasets for metalens endoscopy and conduct preliminary optical simulation, identifying two derived optical issues that physically adhere to strong optical priors. Second, we propose MetaScope, a novel optics-driven neural network tailored for metalens endoscopy driven by physical optics. MetaScope comprises two novel designs: Optics-informed Intensity Adjustment (OIA), rectifying intensity decay by learning optical embeddings, and Optics-informed Chromatic Correction (OCC), mitigating chromatic aberration by learning spatial deformations informed by learned Point Spread Function (PSF) distributions. To enhance joint learning, we further deploy a gradient-guided distillation to transfer knowledge from the foundational model adaptively. Extensive experiments demonstrate that MetaScope not only outperforms state-of-the-art methods in both metalens segmentation and restoration but also achieves impressive generalized ability in real biomedical scenes.