Recovery of Low-Rank Plus Compressed Sparse Matrices with Application to Unveiling Traffic Anomalies

TL;DR

This paper presents a convex optimization approach for exactly recovering low-rank and sparse matrices from compressed superpositions, with applications in traffic anomaly detection, supported by theoretical guarantees and real data tests.

Contribution

It introduces a deterministic condition and a convex program for joint low-rank and sparse matrix recovery from compressed data, applicable to network traffic anomaly detection.

Findings

Convex program successfully recovers matrices under low-rank and sparsity conditions.

High probability of exact recovery when matrices are randomly drawn.

First-order algorithms with provable complexity for solving the optimization problem.

Abstract

Given the superposition of a low-rank matrix plus the product of a known fat compression matrix times a sparse matrix, the goal of this paper is to establish deterministic conditions under which exact recovery of the low-rank and sparse components becomes possible. This fundamental identifiability issue arises with traffic anomaly detection in backbone networks, and subsumes compressed sensing as well as the timely low-rank plus sparse matrix recovery tasks encountered in matrix decomposition problems. Leveraging the ability of $\ell_1$- and nuclear norms to recover sparse and low-rank matrices, a convex program is formulated to estimate the unknowns. Analysis and simulations confirm that the said convex program can recover the unknowns for sufficiently low-rank and sparse enough components, along with a compression matrix possessing an isometry property when restricted to operate on sparse vectors. When the low-rank, sparse, and compression matrices are drawn from certain random ensembles, it is established that exact recovery is possible with high probability. First-order algorithms are developed to solve the nonsmooth convex optimization problem with provable iteration complexity guarantees. Insightful tests with synthetic and real network data corroborate the effectiveness of the novel approach in unveiling traffic anomalies across flows and time, and its ability to outperform existing alternatives.