Efficient Matrix Factorization on Heterogeneous CPU-GPU Systems

TL;DR

This paper presents a novel matrix division and task assignment strategy for stochastic gradient descent-based matrix factorization on heterogeneous CPU-GPU systems, significantly improving computational efficiency.

Contribution

It introduces a new matrix division strategy and a tailored cost model for efficient parallel SGD on heterogeneous systems, balancing workloads and maximizing GPU utilization.

Findings

Achieves high efficiency in MF computation on CPU-GPU systems.

Balances workloads effectively between CPU and GPU.

Maintains high training quality despite optimization.

Abstract

Matrix Factorization (MF) has been widely applied in machine learning and data mining. A large number of algorithms have been studied to factorize matrices. Among them, stochastic gradient descent (SGD) is a commonly used method. Heterogeneous systems with multi-core CPUs and GPUs have become more and more promising recently due to the prevalence of GPUs in general-purpose data-parallel applications. Due to the large computational cost of MF, we aim to improve the efficiency of SGD-based MF computation by utilizing the massive parallel processing power of heterogeneous multiprocessors. The main challenge in parallel SGD algorithms on heterogeneous CPU-GPU systems lies in the granularity of the matrix division and the strategy to assign tasks. We design a novel strategy to divide the matrix into a set of blocks by considering two aspects. First, we observe that the matrix should be divided nonuniformly, and relatively large blocks should be assigned to GPUs to saturate the computing power of GPUs. In addition to exploiting the characteristics of hardware, the workloads assigned to two types of hardware should be balanced. Aiming at the final division strategy, we design a cost model tailored for our problem to accurately estimate the performance of hardware on different data sizes. A dynamic scheduling policy is also used to further balance workloads in practice. Extensive experiments show that our proposed algorithm achieves high efficiency with a high quality of training quality.