Automatic Foreground Extraction from Imperfect Backgrounds using Multi-Agent Consensus Equilibrium

TL;DR

This paper introduces an automatic foreground extraction method that effectively combines alpha matting, background subtraction, and denoising through a novel fusion framework, outperforming existing techniques in complex scenes with imperfect backgrounds.

Contribution

The paper presents a new multi-agent consensus equilibrium framework that integrates multiple methods for improved foreground extraction from imperfect backgrounds.

Findings

Significantly better foreground masks than state-of-the-art methods

Robust performance in complex scenes with imperfect backgrounds

Effective integration of multiple techniques through fusion framework

Abstract

Extracting accurate foreground objects from a scene is an essential step for many video applications. Traditional background subtraction algorithms can generate coarse estimates, but generating high quality masks requires professional softwares with significant human interventions, e.g., providing trimaps or labeling key frames. We propose an automatic foreground extraction method in applications where a static but imperfect background is available. Examples include filming and surveillance where the background can be captured before the objects enter the scene or after they leave the scene. Our proposed method is very robust and produces significantly better estimates than state-of-the-art background subtraction, video segmentation and alpha matting methods. The key innovation of our method is a novel information fusion technique. The fusion framework allows us to integrate the individual strengths of alpha matting, background subtraction and image denoising to produce an overall better estimate. Such integration is particularly important when handling complex scenes with imperfect background. We show how the framework is developed, and how the individual components are built. Extensive experiments and ablation studies are conducted to evaluate the proposed method.