Alternating Direction Method of Multipliers for Nonlinear Matrix Decompositions

TL;DR

This paper introduces an ADMM-based algorithm for nonlinear matrix decompositions that can handle various nonlinear functions and loss metrics, demonstrating broad applicability and efficiency on real datasets.

Contribution

The paper develops a flexible ADMM framework for nonlinear matrix decompositions supporting diverse nonlinearities and loss functions, with demonstrated effectiveness on real-world data.

Findings

Effective for different nonlinear models like ReLU, square, and MinMax functions.

Supports multiple loss functions including least squares, L1, and KL divergence.

Shows promising results on real-world datasets across various applications.

Abstract

We present an algorithm based on the alternating direction method of multipliers (ADMM) for solving nonlinear matrix decompositions (NMD). Given an input matrix $X \in \mathbb{R}^{m \times n}$ and a factorization rank $r \ll \min(m, n)$, NMD seeks matrices $W \in \mathbb{R}^{m \times r}$ and $H \in \mathbb{R}^{r \times n}$ such that $X \approx f(WH)$, where $f$ is an element-wise nonlinear function. We evaluate our method on several representative nonlinear models: the rectified linear unit activation $f(x) = \max(0, x)$, suitable for nonnegative sparse data approximation, the component-wise square $f(x) = x^2$, applicable to probabilistic circuit representation, and the MinMax transform $f(x) = \min(b, \max(a, x))$, relevant for recommender systems. The proposed framework flexibly supports diverse loss functions, including least squares, $\ell_1$ norm, and the Kullback-Leibler divergence, and can be readily extended to other nonlinearities and metrics. We illustrate the applicability, efficiency, and adaptability of the approach on real-world datasets, highlighting its potential for a broad range of applications.