Checkpointing to minimize completion time for Inter-dependent Parallel Processes on Volunteer Grids

TL;DR

This paper develops a mathematical model to optimize checkpoint intervals, reducing completion time for inter-dependent parallel processes in volunteer computing environments, validated through real-world application testing.

Contribution

It introduces a novel mathematical model for determining optimal checkpoint intervals specifically for inter-dependent processes in volunteer grids, enhancing performance.

Findings

Predicted checkpoint intervals closely match empirically optimal ones.

Model effectively minimizes process completion time.

Validation on real-world applications confirms model accuracy.

Abstract

Volunteer computing is being used successfully for large scale scientific computations. This research is in the context of Volpex, a programming framework that supports communicating parallel processes in a volunteer environment. Redundancy and checkpointing are combined to ensure consistent forward progress with Volpex in this unique execution environment characterized by heterogeneous failure prone nodes and interdependent replicated processes. An important parameter for optimizing performance with Volpex is the frequency of checkpointing. The paper presents a mathematical model to minimize the completion time for inter-dependent parallel processes running in a volunteer environment by finding a suitable checkpoint interval. Validation is performed with a sample real world application running on a pool of distributed volunteer nodes. The results indicate that the performance with our predicted checkpoint interval is fairly close to the best performance obtained empirically by varying the checkpoint interval.