Deep Lidar-guided Image Deblurring

TL;DR

This paper explores how integrating Lidar-derived depth data into neural image deblurring models enhances their performance, especially in low-light conditions, by developing a universal adapter and applying continual learning techniques.

Contribution

It introduces a universal adapter structure for depth integration and a continual learning strategy to adapt existing deblurring models with minimal additional data.

Findings

Depth information significantly improves deblurring effectiveness.

The proposed methods outperform baseline models without depth data.

Validation on real-world smartphone Lidar data confirms the approach's practicality.

Abstract

The rise of portable Lidar instruments, including their adoption in smartphones, opens the door to novel computational imaging techniques. Being an active sensing instrument, Lidar can provide complementary data to passive optical sensors, particularly in situations like low-light imaging where motion blur can affect photos. In this paper, we study if the depth information provided by mobile Lidar sensors is useful for the task of image deblurring and how to integrate it with a general approach that transforms any state-of-the-art neural deblurring model into a depth-aware one. To achieve this, we developed a universal adapter structure that efficiently preprocesses the depth information to modulate image features with depth features. Additionally, we applied a continual learning strategy to pretrained encoder-decoder models, enabling them to incorporate depth information as an additional input with minimal extra data requirements. We demonstrate that utilizing true depth information can significantly boost the effectiveness of deblurring algorithms, as validated on a dataset with real-world depth data captured by a smartphone Lidar.