A Partition-insensitive Parallel Framework for Distributed Model Fitting

TL;DR

This paper presents a new parallel framework for distributed model fitting that is insensitive to data partitioning, more efficient, and suitable for high-dimensional data, with proven linear convergence.

Contribution

It introduces a novel parallel framework that overcomes limitations of existing consensus-based methods, enhancing robustness and efficiency in distributed model fitting.

Findings

Framework is insensitive to sample partitioning.

Fewer variables are updated per iteration, increasing efficiency.

Algorithms have a proven worst-case linear convergence rate.

Abstract

Distributed model fitting refers to the process of fitting a mathematical or statistical model to the data using distributed computing resources, such that computing tasks are divided among multiple interconnected computers or nodes, often organized in a cluster or network. Most of the existing methods for distributed model fitting are to formulate it in a consensus optimization problem, and then build up algorithms based on the alternating direction method of multipliers (ADMM). This paper introduces a novel parallel framework for achieving a distributed model fitting. In contrast to previous consensus frameworks, the introduced parallel framework offers two notable advantages. Firstly, it exhibits insensitivity to sample partitioning, meaning that the solution of the algorithm remains unaffected by variations in the number of slave nodes or/and the amount of data each node carries. Secondly, fewer variables are required to be updated at each iteration, so that the proposed parallel framework performs in a more succinct and efficient way, and adapts to high-dimensional data. In addition, we prove that the algorithms under the new parallel framework have a worst-case linear convergence rate in theory. Numerical experiments confirm the generality, robustness, and accuracy of our proposed parallel framework.