A Switched Dynamical System Framework for Analysis of Massively Parallel Asynchronous Numerical Algorithms

TL;DR

This paper introduces a switched dynamical system framework to analyze the stability and accuracy of asynchronous parallel numerical algorithms, addressing scalability issues with new techniques and validating on a GPU-based heat equation simulation.

Contribution

It presents a novel framework modeling asynchronous algorithms as switched dynamical systems and develops scalable analysis techniques for large mode systems.

Findings

Framework effectively analyzes asynchronous algorithms.

Validation on GPU confirms theoretical predictions.

Scalable techniques enable analysis of large mode systems.

Abstract

In the near future, massively parallel computing systems will be necessary to solve computation intensive applications. The key bottleneck in massively parallel implementation of numerical algorithms is the synchronization of data across processing elements (PEs) after each iteration, which results in significant idle time. Thus, there is a trend towards relaxing the synchronization and adopting an asynchronous model of computation to reduce idle time. However, it is not clear what is the effect of this relaxation on the stability and accuracy of the numerical algorithm. In this paper we present a new framework to analyze such algorithms. We treat the computation in each PE as a dynamical system and model the asynchrony as stochastic switching. The overall system is then analyzed as a switched dynamical system. However, modeling of massively parallel numerical algorithms as switched dynamical systems results in a very large number of modes, which makes current analysis tools available for such systems computationally intractable. We develop new techniques that circumvent this scalability issue. The framework is presented on a one-dimensional heat equation as a case study and the proposed analysis framework is verified by solving the partial differential equation (PDE) in a $\mathtt{nVIDIA\: Tesla^{\scriptsize{TM}}}$ GPU machine, with asynchronous communication between cores.