Lightweight Quad Bayer HybridEVS Demosaicing via State Space Augmented Cross-Attention

TL;DR

This paper introduces TSANet, a lightweight two-stage neural network with state space augmented cross-attention for efficient and high-quality demosaicing of HybridEVS event camera data, outperforming previous methods.

Contribution

The paper proposes a novel lightweight two-stage network with state space augmented cross-attention for event-based demosaicing, improving accuracy and efficiency on mobile devices.

Findings

Outperforms DemosaicFormer in PSNR and SSIM across seven datasets.

Reduces parameter count by 1.86 times and computation by 3.29 times.

Demonstrates effective demosaicing on both simulated and real HybridEVS data.

Abstract

Event cameras like the Hybrid Event-based Vision Sensor (HybridEVS) camera capture brightness changes as asynchronous "events" instead of frames, offering advanced application on mobile photography. However, challenges arise from combining a Quad Bayer Color Filter Array (CFA) sensor with event pixels lacking color information, resulting in aliasing and artifacts on the demosaicing process before downstream application. Current methods struggle to address these issues, especially on resource-limited mobile devices. In response, we introduce \textbf{TSANet}, a lightweight \textbf{T}wo-stage network via \textbf{S}tate space augmented cross-\textbf{A}ttention, which can handle event pixels inpainting and demosaicing separately, leveraging the benefits of dividing complex tasks into manageable subtasks. Furthermore, we introduce a lightweight Cross-Swin State Block that uniquely utilizes positional prior for demosaicing and enhances global dependencies through the state space model with linear complexity. In summary, TSANet demonstrates excellent demosaicing performance on both simulated and real data of HybridEVS while maintaining a lightweight model, averaging better results than the previous state-of-the-art method DemosaicFormer across seven diverse datasets in both PSNR and SSIM, while respectively reducing parameter and computation costs by $1.86\times$ and $3.29\times$. Our approach presents new possibilities for efficient image demosaicing on mobile devices. Code is available in the supplementary materials.