# Distributed Stochastic Gradient Method for Non-Convex Problems with Applications in Supervised Learning

## TL;DR

This paper introduces a distributed stochastic gradient descent algorithm tailored for non-convex optimization problems, demonstrating its effectiveness in collaborative neural network training for digit recognition across networked agents.

## Contribution

It presents a novel distributed stochastic gradient method with convergence guarantees for non-convex problems and applies it successfully to distributed supervised learning tasks.

## Key findings

- Agents achieve similar performance to centralized training
- Distributed training enables recognition without local data for all classes
- Algorithm converges under specific step-size conditions

## Abstract

We develop a distributed stochastic gradient descent algorithm for solving non-convex optimization problems under the assumption that the local objective functions are twice continuously differentiable with Lipschitz continuous gradients and Hessians. We provide sufficient conditions on step-sizes that guarantee the asymptotic mean-square convergence of the proposed algorithm. We apply the developed algorithm to a distributed supervised-learning problem, in which a set of networked agents collaboratively train their individual neural nets to recognize handwritten digits in images. Results indicate that all agents report similar performance that is also comparable to the performance of a centrally trained neural net. Numerical results also show that the proposed distributed algorithm allows the individual agents to recognize the digits even though the training data corresponding to all the digits is not locally available to each agent.