PSO-PS: Parameter Synchronization with Particle Swarm Optimization for Distributed Training of Deep Neural Networks

TL;DR

This paper introduces a novel distributed deep learning training method that uses Particle Swarm Optimization to synchronize parameters, reducing communication overhead while maintaining good convergence.

Contribution

The paper proposes integrating PSO into distributed DNN training to automatically optimize parameters at synchronization points, improving over traditional averaging methods.

Findings

Achieves competitive accuracy on MNIST and CIFAR10.

Reduces synchronization frequency without losing convergence.

Demonstrates effectiveness compared to peer methods.

Abstract

Parameter updating is an important stage in parallelism-based distributed deep learning. Synchronous methods are widely used in distributed training the Deep Neural Networks (DNNs). To reduce the communication and synchronization overhead of synchronous methods, decreasing the synchronization frequency (e.g., every $n$ mini-batches) is a straightforward approach. However, it often suffers from poor convergence. In this paper, we propose a new algorithm of integrating Particle Swarm Optimization (PSO) into the distributed training process of DNNs to automatically compute new parameters. In the proposed algorithm, a computing work is encoded by a particle, the weights of DNNs and the training loss are modeled by the particle attributes. At each synchronization stage, the weights are updated by PSO from the sub weights gathered from all workers, instead of averaging the weights or the gradients. To verify the performance of the proposed algorithm, the experiments are performed on two commonly used image classification benchmarks: MNIST and CIFAR10, and compared with the peer competitors at multiple different synchronization configurations. The experimental results demonstrate the competitiveness of the proposed algorithm.