Adaptive Mask-based Pyramid Network for Realistic Bokeh Rendering

TL;DR

This paper introduces AMPN, a lightweight, adaptive pyramid network for mobile-friendly, high-resolution bokeh rendering that allows user-guided mask editing and varying blur strength, outperforming existing methods in speed and size.

Contribution

We propose the Adaptive Mask-based Pyramid Network (AMPN), combining a mask-guided generator and pyramid refinement for efficient, high-quality, user-controllable bokeh rendering on mobile devices.

Findings

AMPN achieves high-resolution bokeh with on-device performance.

It allows user-guided mask editing for customizable effects.

AMPN outperforms existing methods in speed and model size.

Abstract

Bokeh effect highlights an object (or any part of the image) while blurring the rest of the image, and creates a visually pleasant artistic effect. Due to the sensor-based limitations on mobile devices, machine learning (ML) based bokeh rendering has gained attention as a reliable alternative. In this paper, we focus on several improvements in ML-based bokeh rendering; i) on-device performance with high-resolution images, ii) ability to guide bokeh generation with user-editable masks and iii) ability to produce varying blur strength. To this end, we propose Adaptive Mask-based Pyramid Network (AMPN), which is formed of a Mask-Guided Bokeh Generator (MGBG) block and a Laplacian Pyramid Refinement (LPR) block. MGBG consists of two lightweight networks stacked to each other to generate the bokeh effect, and LPR refines and upsamples the output of MGBG to produce the high-resolution bokeh image. We achieve i) via our lightweight, mobile-friendly design choices, ii) via the stacked-network design of MGBG and the weakly-supervised mask prediction scheme and iii) via manually or automatically editing the intensity values of the mask that guide the bokeh generation. In addition to these features, our results show that AMPN produces competitive or better results compared to existing methods on the EBB! dataset, while being faster and smaller than the alternatives.