Support matrix machine: exploring sample sparsity, low rank, and adaptive sieving in high-performance computing

TL;DR

This paper introduces an efficient method for large-scale support matrix machine training that leverages sample sparsity, low-rank regularization, and adaptive sieving to reduce computational costs and improve convergence.

Contribution

It develops a semismooth Newton-CG based augmented Lagrangian method with adaptive sieving for scalable support matrix machine optimization.

Findings

Achieves superlinear convergence under certain conditions

Reduces computational and storage costs significantly

Validates effectiveness on large-scale datasets

Abstract

Support matrix machine (SMM) is a successful supervised classification model for matrix-type samples. Unlike support vector machines, it employs low-rank regularization on the regression matrix to effectively capture the intrinsic structure embedded in each input matrix. When solving a large-scale SMM, a major challenge arises from the potential increase in sample size, leading to substantial computational and storage burdens. To address these issues, we design a semismooth Newton-CG (SNCG) based augmented Lagrangian method (ALM) for solving the SMM. The ALM exhibits an asymptotic R-superlinear convergence if a strict complementarity condition is satisfied. The SNCG method is employed to solve the ALM subproblems, achieving at least a superlinear convergence rate under the nonemptiness of an index set. Furthermore, the sparsity of samples and the low-rank nature of solutions enable us to reduce the computational cost and storage demands for the Newton linear systems. Additionally, we develop an adaptive sieving strategy that generates a solution path for the SMM by exploiting sample sparsity. The finite convergence of this strategy is also demonstrated. Numerical experiments on both large-scale real and synthetic datasets validate the effectiveness of the proposed methods.