# Distributed Gradient Descent: Nonconvergence to Saddle Points and the Stable-Manifold Theorem

## TL;DR

This paper extends the stable-manifold theorem to distributed gradient descent, demonstrating that under certain conditions, DGD almost always converges to local minima rather than saddle points, addressing a key challenge in nonconvex optimization.

## Contribution

It develops a novel stable-manifold theorem tailored for distributed gradient descent, showing convergence to saddle points is highly unlikely in nonconvex problems.

## Key findings

- DGD typically converges to local minima, not saddle points
- Convergence to saddle points occurs only on a low-dimensional stable manifold
- Under certain assumptions, DGD almost always avoids saddle points

## Abstract

The paper studies a distributed gradient descent (DGD) process and considers the problem of showing that in nonconvex optimization problems, DGD typically converges to local minima rather than saddle points. The paper considers unconstrained minimization of a smooth objective function. In centralized settings, the problem of demonstrating nonconvergence to saddle points of gradient descent (and variants) is typically handled by way of the stable-manifold theorem from classical dynamical systems theory. However, the classical stable-manifold theorem is not applicable in distributed settings. The paper develops an appropriate stable-manifold theorem for DGD showing that convergence to saddle points may only occur from a low-dimensional stable manifold. Under appropriate assumptions (e.g., coercivity), this result implies that DGD typically converges to local minima and not to saddle points.