Performance Evaluation of an Algorithm-based Asynchronous Checkpoint-Restart Fault Tolerant Application Using Mixed MPI/GPI-2

TL;DR

This paper compares the performance and fault tolerance of a traditional MPI-based approach with a GPI-2 based asynchronous checkpoint-restart method for running K-means on large datasets in a distributed cluster.

Contribution

It introduces and evaluates a fault-tolerant, asynchronous checkpoint-restart algorithm using GPI-2, contrasting it with conventional MPI and Spark solutions for big data processing.

Findings

GPI-2 based method shows improved fault tolerance.

MPI approach has longer recovery times.

GPI-2 method reduces computational time for large datasets.

Abstract

One of the hardest challenges of the current Big Data landscape is the lack of ability to process huge volumes of information in an acceptable time. The goal of this work, is to ascertain if it is useful to use typical Big Data tools to solve High Performance Computing problems, by exploring and comparing a distributed computing framework implemented on a commodity cluster architecture: the experiment will depend on the computational time required using tools such as Apache Spark. This will be compared to "equivalent more traditional" approaches such as using a distributed memory model with MPI on a distributed file system such as HDFS (Hadoop Distributed File System) and native C libraries that create an interface to encapsulate this file system functionalities, and using the GPI-2 implementation for the GASPI protocol and it's in-memory checkpointing library to provide an application with Fault Tolerance features. To be more precise, we've chosen the K-means algorithm as experiment, that will be ran on variable size datasets, and then we will compare the computational run time and time resilience of both approaches.