Deep learning based fence segmentation and removal from an image using a video sequence

TL;DR

This paper introduces a novel deep learning-based method for fence segmentation and removal from images in dynamic scenes using video sequences, combining CNN-based segmentation, occlusion-aware optical flow, and data fusion within an optimization framework.

Contribution

It presents a new approach that extends de-fencing to dynamic scenes by integrating CNN segmentation with occlusion-aware optical flow and data fusion, unlike prior static scene methods.

Findings

Effective fence segmentation from a single image using CNN.

Successful removal of fences in dynamic scenes with occlusions.

Improved image quality after fence removal demonstrated experimentally.

Abstract

Conventional approaches to image de-fencing use multiple adjacent frames for segmentation of fences in the reference image and are limited to restoring images of static scenes only. In this paper, we propose a de-fencing algorithm for images of dynamic scenes using an occlusion-aware optical flow method. We divide the problem of image de-fencing into the tasks of automated fence segmentation from a single image, motion estimation under known occlusions and fusion of data from multiple frames of a captured video of the scene. Specifically, we use a pre-trained convolutional neural network to segment fence pixels from a single image. The knowledge of spatial locations of fences is used to subsequently estimate optical flow in the occluded frames of the video for the final data fusion step. We cast the fence removal problem in an optimization framework by modeling the formation of the degraded observations. The inverse problem is solved using fast iterative shrinkage thresholding algorithm (FISTA). Experimental results show the effectiveness of proposed algorithm.