GANESH: Generalizable NeRF for Lensless Imaging

TL;DR

GANESH introduces a generalizable framework for lensless imaging that enables high-quality 3D reconstruction and view synthesis without scene-specific training, supported by a new multi-view dataset.

Contribution

The paper presents GANESH, a novel method for lensless imaging that generalizes across scenes and allows on-the-fly inference and scene tuning, advancing beyond prior scene-specific approaches.

Findings

Outperforms existing methods in reconstruction accuracy

Supports real-time inference without retraining

Provides the first multi-view lensless dataset

Abstract

Lensless imaging offers a significant opportunity to develop ultra-compact cameras by removing the conventional bulky lens system. However, without a focusing element, the sensor's output is no longer a direct image but a complex multiplexed scene representation. Traditional methods have attempted to address this challenge by employing learnable inversions and refinement models, but these methods are primarily designed for 2D reconstruction and do not generalize well to 3D reconstruction. We introduce GANESH, a novel framework designed to enable simultaneous refinement and novel view synthesis from multi-view lensless images. Unlike existing methods that require scene-specific training, our approach supports on-the-fly inference without retraining on each scene. Moreover, our framework allows us to tune our model to specific scenes, enhancing the rendering and refinement quality. To facilitate research in this area, we also present the first multi-view lensless dataset, LenslessScenes. Extensive experiments demonstrate that our method outperforms current approaches in reconstruction accuracy and refinement quality. Code and video results are available at https://rakesh-123-cryp.github.io/Rakesh.github.io/