Distributed Weighted Parameter Averaging for SVM Training on Big Data

TL;DR

This paper introduces a hybrid weighted parameter averaging method for distributed SVM training that improves accuracy and convergence speed over traditional approaches, especially with many data partitions.

Contribution

The paper proposes WPA, a novel hybrid method that combines parameter averaging and ADMM, with theoretical stability bounds and improved empirical performance.

Findings

WPA achieves higher accuracy than simple parameter averaging with many partitions.

WPA converges faster than ADMM in feature space.

Experimental results confirm the effectiveness of WPA on real datasets.

Abstract

Two popular approaches for distributed training of SVMs on big data are parameter averaging and ADMM. Parameter averaging is efficient but suffers from loss of accuracy with increase in number of partitions, while ADMM in the feature space is accurate but suffers from slow convergence. In this paper, we report a hybrid approach called weighted parameter averaging (WPA), which optimizes the regularized hinge loss with respect to weights on parameters. The problem is shown to be same as solving SVM in a projected space. We also demonstrate an $O(\frac{1}{N})$ stability bound on final hypothesis given by WPA, using novel proof techniques. Experimental results on a variety of toy and real world datasets show that our approach is significantly more accurate than parameter averaging for high number of partitions. It is also seen the proposed method enjoys much faster convergence compared to ADMM in features space.