Multimodal Unrolled Robust PCA for Background Foreground Separation

TL;DR

This paper enhances background foreground separation by integrating radar data with Robust PCA, using algorithm unrolling for real-time, robust, and generalizable performance, especially under challenging conditions.

Contribution

It introduces a novel multimodal approach combining radar and Robust PCA, employing algorithm unrolling for improved robustness and real-time inference in BFS tasks.

Findings

Quantitative improvements with radar data integration.

Robustness to lighting changes, reflections, and occlusion.

Real-time, feedforward inference achieved.

Abstract

Background foreground separation (BFS) is a popular computer vision problem where dynamic foreground objects are separated from the static background of a scene. Typically, this is performed using consumer cameras because of their low cost, human interpretability, and high resolution. Yet, cameras and the BFS algorithms that process their data have common failure modes due to lighting changes, highly reflective surfaces, and occlusion. One solution is to incorporate an additional sensor modality that provides robustness to such failure modes. In this paper, we explore the ability of a cost-effective radar system to augment the popular Robust PCA technique for BFS. We apply the emerging technique of algorithm unrolling to yield real-time computation, feedforward inference, and strong generalization in comparison with traditional deep learning methods. We benchmark on the RaDICaL dataset to demonstrate both quantitative improvements of incorporating radar data and qualitative improvements that confirm robustness to common failure modes of image-based methods.