Efficient coordinate-descent for orthogonal matrices through Givens rotations

TL;DR

This paper introduces a Givens-rotation based coordinate descent framework for efficiently optimizing orthogonal matrices, enabling faster convergence and better models in applications like tensor decomposition and sparse-PCA.

Contribution

It presents a novel Givens-rotation based coordinate descent method for orthogonal matrix optimization, improving computational efficiency and convergence in relevant applications.

Findings

Faster convergence in tensor decomposition and sparse-PCA tasks.

Superior model performance on genome-wide mRNA expression data.

Efficient preservation of orthogonality during optimization.

Abstract

Optimizing over the set of orthogonal matrices is a central component in problems like sparse-PCA or tensor decomposition. Unfortunately, such optimization is hard since simple operations on orthogonal matrices easily break orthogonality, and correcting orthogonality usually costs a large amount of computation. Here we propose a framework for optimizing orthogonal matrices, that is the parallel of coordinate-descent in Euclidean spaces. It is based on {\em Givens-rotations}, a fast-to-compute operation that affects a small number of entries in the learned matrix, and preserves orthogonality. We show two applications of this approach: an algorithm for tensor decomposition that is used in learning mixture models, and an algorithm for sparse-PCA. We study the parameter regime where a Givens rotation approach converges faster and achieves a superior model on a genome-wide brain-wide mRNA expression dataset.