Simultaneous penalized M-estimation of covariance matrices using geodesically convex optimization

TL;DR

This paper introduces two novel penalized M-estimation methods for covariance matrices that allow for deviations among groups, utilizing geodesic convexity to ensure unique solutions and applying them to regularized discriminant analysis.

Contribution

It proposes two new approaches for estimating multiple covariance matrices with shared structure, leveraging geodesic convexity for theoretical guarantees.

Findings

Existence and uniqueness of solutions are proven under general conditions.

The methods relate to means of positive definite matrices like arithmetic and harmonic.

Application to regularized discriminant analysis demonstrates practical benefits.

Abstract

A common assumption when sampling $p$-dimensional observations from $K$ distinct group is the equality of the covariance matrices. In this paper, we propose two penalized $M$-estimation approaches for the estimation of the covariance or scatter matrices under the broader assumption that they may simply be close to each other, and hence roughly deviate from some positive definite "center". The first approach begins by generating a pooled $M$-estimator of scatter based on all the data, followed by a penalised $M$-estimator of scatter for each group, with the penalty term chosen so that the individual scatter matrices are shrunk towards the pooled scatter matrix. In the second approach, we minimize the sum of the individual group $M$-estimation cost functions together with an additive joint penalty term which enforces some similarity between the individual scatter estimators, i.e. shrinkage towards a mutual center. In both approaches, we utilize the concept of geodesic convexity to prove the existence and uniqueness of the penalized solution under general conditions. We consider three specific penalty functions based on the Euclidean, the Riemannian, and the Kullback-Leibler distances. In the second approach, the distance based penalties are shown to lead to estimators of the mutual center that are related to the arithmetic, the Riemannian and the harmonic means of positive definite matrices, respectively. A penalty based on an ellipticity measure is also considered which is particularly useful for shape matrix estimators. Fixed point equations are derived for each penalty function and the benefits of the estimators are illustrated in regularized discriminant analysis problem.