PolarAPP: Beyond Polarization Demosaicking for Polarimetric Applications

TL;DR

PolarAPP is a novel framework that jointly optimizes polarimetric image reconstruction and downstream tasks, improving accuracy by making demosaicking task-aware through meta-learning and physical constraints.

Contribution

It introduces the first joint optimization approach for demosaicking and downstream polarimetric applications, incorporating semantic feature alignment and physical constraints.

Findings

Outperforms existing methods in demosaicking quality.

Enhances downstream task performance significantly.

Demonstrates the effectiveness of task-aware reconstruction.

Abstract

Polarimetric imaging enables advanced vision applications such as normal estimation and de-reflection by capturing unique surface-material interactions. However, existing applications (alternatively called downstream tasks) rely on datasets constructed by naively regrouping raw measurements from division-of-focal-plane sensors, where pixels of the same polarization angle are extracted and aligned into sparse images without proper demosaicking. This reconstruction strategy results in suboptimal, incomplete targets that limit downstream performance. Moreover, current demosaicking methods are task-agnostic, optimizing only for photometric fidelity rather than utility in downstream tasks. Towards this end, we propose PolarAPP, the first framework to jointly optimize demosaicking and its downstream tasks. PolarAPP introduces a feature alignment mechanism that semantically aligns the representations of demosaicking and downstream networks via meta-learning, guiding the reconstruction to be task-aware. It further employs an equivalent imaging constraint for demosaicking training, enabling direct regression to physically meaningful outputs without relying on rearranged data. Finally, a task-refinement stage fine-tunes the task network using the stable demosaicking front-end to further enhance accuracy. Extensive experimental results demonstrate that PolarAPP outperforms existing methods in both demosaicking quality and downstream performance. Code is available upon acceptance.