Iterative Thresholding Algorithm for Sparse Inverse Covariance Estimation

TL;DR

This paper introduces G-ISTA, a proximal gradient algorithm for sparse inverse covariance estimation that converges linearly and performs well especially with well-conditioned solutions.

Contribution

The paper proposes G-ISTA, a simple yet effective proximal gradient method with proven linear convergence for sparse inverse covariance estimation.

Findings

G-ISTA has a linear convergence rate with O(log e) iteration complexity.

Eigenvalue bounds for G-ISTA iterates are provided.

Numerical results show G-ISTA performs well, especially with well-conditioned solutions.

Abstract

The L1-regularized maximum likelihood estimation problem has recently become a topic of great interest within the machine learning, statistics, and optimization communities as a method for producing sparse inverse covariance estimators. In this paper, a proximal gradient method (G-ISTA) for performing L1-regularized covariance matrix estimation is presented. Although numerous algorithms have been proposed for solving this problem, this simple proximal gradient method is found to have attractive theoretical and numerical properties. G-ISTA has a linear rate of convergence, resulting in an O(log e) iteration complexity to reach a tolerance of e. This paper gives eigenvalue bounds for the G-ISTA iterates, providing a closed-form linear convergence rate. The rate is shown to be closely related to the condition number of the optimal point. Numerical convergence results and timing comparisons for the proposed method are presented. G-ISTA is shown to perform very well, especially when the optimal point is well-conditioned.