Dynamic Mode Decomposition for Real-Time Background/Foreground Separation in Video

TL;DR

This paper presents a real-time, computationally efficient method using dynamic mode decomposition (DMD) to separate video into background and foreground components, outperforming existing techniques like RPCA.

Contribution

The paper introduces a novel application of DMD for real-time background/foreground separation in video, achieving high speed without parameter tuning.

Findings

DMD-based separation is orders of magnitude faster than RPCA.

The method works robustly on personal laptops in real-time.

No parameter tuning required for the DMD approach.

Abstract

This paper introduces the method of dynamic mode decomposition (DMD) for robustly separating video frames into background (low-rank) and foreground (sparse) components in real-time. The method is a novel application of a technique used for characterizing nonlinear dynamical systems in an equation-free manner by decomposing the state of the system into low-rank terms whose Fourier components in time are known. DMD terms with Fourier frequencies near the origin (zero-modes) are interpreted as background (low-rank) portions of the given video frames, and the terms with Fourier frequencies bounded away from the origin are their sparse counterparts. An approximate low-rank/sparse separation is achieved at the computational cost of just one singular value decomposition and one linear equation solve, thus producing results orders of magnitude faster than a leading separation method, namely robust principal component analysis (RPCA). The DMD method that is developed here is demonstrated to work robustly in real-time with personal laptop-class computing power and without any parameter tuning, which is a transformative improvement in performance that is ideal for video surveillance and recognition applications.