On the Trade-off between Flatness and Optimization in Distributed Learning

TL;DR

This paper develops a theoretical framework to analyze how decentralized learning strategies influence the trade-off between flatness of minima and optimization in nonconvex distributed learning, showing decentralized methods often outperform centralized ones in classification accuracy.

Contribution

It introduces a novel theoretical analysis of decentralized learning, revealing how strategies like diffusion better balance flatness and optimization, leading to improved accuracy.

Findings

Decentralized strategies escape local minima faster and favor flatter minima.

Diffusion outperforms consensus in excess-risk, aiding in escaping local minima.

Classification accuracy depends on both flatness and optimization performance.

Abstract

This paper proposes a theoretical framework to evaluate and compare the performance of stochastic gradient algorithms for distributed learning in relation to their behavior around local minima in nonconvex environments. Previous works have noticed that convergence toward flat local minima tend to enhance the generalization ability of learning algorithms. This work discovers three interesting results. First, it shows that decentralized learning strategies are able to escape faster away from local minima and favor convergence toward flatter minima relative to the centralized solution. Second, in decentralized methods, the consensus strategy has a worse excess-risk performance than diffusion, giving it a better chance of escaping from local minima and favoring flatter minima. Third, and importantly, the ultimate classification accuracy is not solely dependent on the flatness of the local minimum but also on how well a learning algorithm can approach that minimum. In other words, the classification accuracy is a function of both flatness and optimization performance. In this regard, since diffusion has a lower excess-risk than consensus, when both algorithms are trained starting from random initial points, diffusion enhances the classification accuracy. The paper examines the interplay between the two measures of flatness and optimization error closely. One important conclusion is that decentralized strategies deliver in general enhanced classification accuracy because they strike a more favorable balance between flatness and optimization performance compared to the centralized solution.