Fovea Stacking: Imaging with Dynamic Localized Aberration Correction

TL;DR

Fovea Stacking introduces a novel imaging system using deformable phase plates to correct aberrations locally, creating high-quality, foveated images that can be combined for wide-field, aberration-free imaging, with applications in real-time video.

Contribution

The paper presents a new foveated imaging system with dynamic aberration correction using deformable phase plates and joint optimization, advancing computational imaging with adaptive, localized optical correction.

Findings

Fovea stacking outperforms traditional focus stacking in extended depth-of-field imaging.

The system enables real-time, foveated video with dynamic object tracking.

Neural network control improves alignment between simulation and hardware performance.

Abstract

The desire for cameras with smaller form factors has recently lead to a push for exploring computational imaging systems with reduced optical complexity such as a smaller number of lens elements. Unfortunately such simplified optical systems usually suffer from severe aberrations, especially in off-axis regions, which can be difficult to correct purely in software. In this paper we introduce Fovea Stacking , a new type of imaging system that utilizes emerging dynamic optical components called deformable phase plates (DPPs) for localized aberration correction anywhere on the image sensor. By optimizing DPP deformations through a differentiable optical model, off-axis aberrations are corrected locally, producing a foveated image with enhanced sharpness at the fixation point - analogous to the eye's fovea. Stacking multiple such foveated images, each with a different fixation point, yields a composite image free from aberrations. To efficiently cover the entire field of view, we propose joint optimization of DPP deformations under imaging budget constraints. Due to the DPP device's non-linear behavior, we introduce a neural network-based control model for improved alignment between simulation-hardware performance. We further demonstrated that for extended depth-of-field imaging, fovea stacking outperforms traditional focus stacking in image quality. By integrating object detection or eye-tracking, the system can dynamically adjust the lens to track the object of interest-enabling real-time foveated video suitable for downstream applications such as surveillance or foveated virtual reality displays