# Scaling Up Quasi-Newton Algorithms: Communication Efficient Distributed SR1

## TL;DR

This paper introduces DS-LSR1, a communication-efficient distributed quasi-Newton algorithm that significantly reduces communication overhead and scales effectively for large neural network training tasks.

## Contribution

The paper proposes DS-LSR1, a novel distributed implementation of S-LSR1 that is communication-efficient, matrix-free, inverse-free, and scalable for high-dimensional problems.

## Key findings

- Reduces communication rounds in distributed S-LSR1
- Achieves better load balancing across nodes
- Demonstrates effective scaling on neural network training

## Abstract

In this paper, we present a scalable distributed implementation of the Sampled Limited-memory Symmetric Rank-1 (S-LSR1) algorithm. First, we show that a naive distributed implementation of S-LSR1 requires multiple rounds of expensive communications at every iteration and thus is inefficient. We then propose DS-LSR1, a communication-efficient variant that: (i) drastically reduces the amount of data communicated at every iteration, (ii) has favorable work-load balancing across nodes, and (iii) is matrix-free and inverse-free. The proposed method scales well in terms of both the dimension of the problem and the number of data points. Finally, we illustrate the empirical performance of DS-LSR1 on a standard neural network training task.