Regularized $M$-estimators of scatter matrix

TL;DR

This paper introduces a new class of regularized M-estimators for scatter matrices, suitable for small sample sizes, with theoretical guarantees, practical algorithms, and demonstrated improvements in radar detection tasks.

Contribution

It generalizes M-estimators with regularization, derives conditions for solution uniqueness, and provides a practical data-driven regularization method and iterative algorithm.

Findings

Regularized estimators outperform non-regularized in low sample scenarios.

Closed-form regularization parameter selection improves practical applicability.

Enhanced radar detection performance with adaptive NMF using regularized estimators.

Abstract

In this paper, a general class of regularized $M$-estimators of scatter matrix are proposed which are suitable also for low or insufficient sample support (small $n$ and large $p$) problems. The considered class constitutes a natural generalization of $M$-estimators of scatter matrix (Maronna, 1976) and are defined as a solution to a penalized $M$-estimation cost function that depend on a pair $(\alpha,\beta)$ of regularization parameters. We derive general conditions for uniqueness of the solution using concept of geodesic convexity. Since these conditions do not include Tyler's $M$-estimator, necessary and sufficient conditions for uniqueness of the penalized Tyler's cost function are established separately. For the regularized Tyler's $M$-estimator, we also derive a simple, closed form and data dependent solution for choosing the regularization parameter based on shape matrix matching in the mean squared sense. An iterative algorithm that converges to the solution of the regularized $M$-estimating equation is also provided. Finally, some simulations studies illustrate the improved accuracy of the proposed regularized $M$-estimators of scatter compared to their non-regularized counterparts in low sample support problems. An example of radar detection using normalized matched filter (NMF) illustrate that an adaptive NMF detector based on regularized $M$-estimators are able to maintain accurately the preset CFAR level and at at the same time provide similar probability of detection as the (theoretical) NMF detector.