Augmented Robust PCA For Foreground-Background Separation on Noisy, Moving Camera Video

TL;DR

This paper introduces an augmented robust PCA method with total variation regularization for effective foreground-background separation and denoising in noisy, moving camera videos, leveraging frame registration and a novel low-rank estimator.

Contribution

It presents a new robust PCA algorithm that registers frames, produces panoramic low-rank backgrounds, and uses OptShrink for parameter-free low-rank estimation in challenging video scenarios.

Findings

Effective separation of foreground and background in noisy, moving camera videos.

Automatic stitching of corrupted data into a panoramic low-rank background.

Robust performance demonstrated on videos with noise and outliers.

Abstract

This work presents a novel approach for robust PCA with total variation regularization for foreground-background separation and denoising on noisy, moving camera video. Our proposed algorithm registers the raw (possibly corrupted) frames of a video and then jointly processes the registered frames to produce a decomposition of the scene into a low-rank background component that captures the static components of the scene, a smooth foreground component that captures the dynamic components of the scene, and a sparse component that can isolate corruptions and other non-idealities. Unlike existing methods, our proposed algorithm produces a panoramic low-rank component that spans the entire field of view, automatically stitching together corrupted data from partially overlapping scenes. The low-rank portion of our robust PCA model is based on a recently discovered optimal low-rank matrix estimator (OptShrink) that requires no parameter tuning. We demonstrate the performance of our algorithm on both static and moving camera videos corrupted by noise and outliers.