Degradation-Modeled Multipath Diffusion for Tunable Metalens Photography

TL;DR

This paper presents a novel diffusion-based framework for metalens imaging that uses natural image priors and adaptive degradation modeling to improve image quality and control over artifacts, validated on a real MetaCamera.

Contribution

Introduces Degradation-Modeled Multipath Diffusion, combining prompt paths and a degradation-aware attention module for enhanced metalens image reconstruction without large datasets.

Findings

Outperforms state-of-the-art methods in image fidelity and sharpness.

Enables controlled trade-offs between image quality and perceptual realism.

Validated on a custom-built millimeter-scale MetaCamera.

Abstract

Metalenses offer significant potential for ultra-compact computational imaging but face challenges from complex optical degradation and computational restoration difficulties. Existing methods typically rely on precise optical calibration or massive paired datasets, which are non-trivial for real-world imaging systems. Furthermore, a lack of control over the inference process often results in undesirable hallucinated artifacts. We introduce Degradation-Modeled Multipath Diffusion for tunable metalens photography, leveraging powerful natural image priors from pretrained models instead of large datasets. Our framework uses positive, neutral, and negative-prompt paths to balance high-frequency detail generation, structural fidelity, and suppression of metalens-specific degradation, alongside \textit{pseudo} data augmentation. A tunable decoder enables controlled trade-offs between fidelity and perceptual quality. Additionally, a spatially varying degradation-aware attention (SVDA) module adaptively models complex optical and sensor-induced degradation. Finally, we design and build a millimeter-scale MetaCamera for real-world validation. Extensive results show that our approach outperforms state-of-the-art methods, achieving high-fidelity and sharp image reconstruction. More materials: https://dmdiff.github.io/.