High-Dimensional Sparse Data Low-rank Representation via Accelerated Asynchronous Parallel Stochastic Gradient Descent

TL;DR

This paper introduces A2PSGD, an accelerated asynchronous parallel stochastic gradient descent method, to efficiently compute low-rank representations of high-dimensional sparse data, significantly improving convergence speed and accuracy.

Contribution

It proposes a novel optimization algorithm combining lock-free scheduling, load balancing, and Nesterov acceleration for high-dimensional sparse data low-rank modeling.

Findings

A2PSGD outperforms existing algorithms in accuracy.

A2PSGD reduces training time significantly.

The method effectively handles large-scale high-dimensional sparse data.

Abstract

Data characterized by high dimensionality and sparsity are commonly used to describe real-world node interactions. Low-rank representation (LR) can map high-dimensional sparse (HDS) data to low-dimensional feature spaces and infer node interactions via modeling data latent associations. Unfortunately, existing optimization algorithms for LR models are computationally inefficient and slowly convergent on large-scale datasets. To address this issue, this paper proposes an Accelerated Asynchronous Parallel Stochastic Gradient Descent A2PSGD for High-Dimensional Sparse Data Low-rank Representation with three fold-ideas: a) establishing a lock-free scheduler to simultaneously respond to scheduling requests from multiple threads; b) introducing a greedy algorithm-based load balancing strategy for balancing the computational load among threads; c) incorporating Nesterov's accelerated gradient into the learning scheme to accelerate model convergence. Empirical studies show that A2PSGD outperforms existing optimization algorithms for HDS data LR in both accuracy and training time.