Computational Imaging for Machine Perception: Transferring Semantic Segmentation beyond Aberrations

TL;DR

This paper explores semantic segmentation in optical systems affected by aberrations, proposing a novel domain adaptation method that enhances robustness and bridges the gap between computational imaging and perception tasks.

Contribution

It introduces a new benchmark for semantic segmentation under aberrations and proposes CIADA, a domain adaptation technique leveraging computational imaging for improved performance.

Findings

CIADA outperforms classical segmenters across aberration levels

Benchmark datasets demonstrate the impact of aberrations on segmentation accuracy

Extensive evaluations validate the robustness of the proposed method

Abstract

Semantic scene understanding with Minimalist Optical Systems (MOS) in mobile and wearable applications remains a challenge due to the corrupted imaging quality induced by optical aberrations. However, previous works only focus on improving the subjective imaging quality through the Computational Imaging (CI) technique, ignoring the feasibility of advancing semantic segmentation. In this paper, we pioneer the investigation of Semantic Segmentation under Optical Aberrations (SSOA) with MOS. To benchmark SSOA, we construct Virtual Prototype Lens (VPL) groups through optical simulation, generating Cityscapes-ab and KITTI-360-ab datasets under different behaviors and levels of aberrations. We look into SSOA via an unsupervised domain adaptation perspective to address the scarcity of labeled aberration data in real-world scenarios. Further, we propose Computational Imaging Assisted Domain Adaptation (CIADA) to leverage prior knowledge of CI for robust performance in SSOA. Based on our benchmark, we conduct experiments on the robustness of classical segmenters against aberrations. In addition, extensive evaluations of possible solutions to SSOA reveal that CIADA achieves superior performance under all aberration distributions, bridging the gap between computational imaging and downstream applications for MOS. The project page is at https://github.com/zju-jiangqi/CIADA.