SteReFo: Efficient Image Refocusing with Stereo Vision

TL;DR

SteReFo is a real-time, differentiable pipeline that uses stereo vision to efficiently simulate shallow depth of field effects, enabling dynamic refocusing and video focus tracking from all-in-focus images.

Contribution

It introduces a physically motivated, scene-agnostic, and fully differentiable stereo-based refocusing pipeline capable of real-time performance and dynamic focus adjustment.

Findings

Achieves 76 FPS on KITTI images for real-time refocusing.

Works effectively on multiple datasets including SceneFlow, KITTI, and CityScapes.

Supports computational video focus tracking for moving objects.

Abstract

Whether to attract viewer attention to a particular object, give the impression of depth or simply reproduce human-like scene perception, shallow depth of field images are used extensively by professional and amateur photographers alike. To this end, high quality optical systems are used in DSLR cameras to focus on a specific depth plane while producing visually pleasing bokeh. We propose a physically motivated pipeline to mimic this effect from all-in-focus stereo images, typically retrieved by mobile cameras. It is capable to change the focal plane a posteriori at 76 FPS on KITTI images to enable real-time applications. As our portmanteau suggests, SteReFo interrelates stereo-based depth estimation and refocusing efficiently. In contrast to other approaches, our pipeline is simultaneously fully differentiable, physically motivated, and agnostic to scene content. It also enables computational video focus tracking for moving objects in addition to refocusing of static images. We evaluate our approach on the publicly available datasets SceneFlow, KITTI, CityScapes and quantify the quality of architectural changes.