# Application-Level Differential Checkpointing for HPC Applications with Dynamic Datasets

## TL;DR

This paper introduces a novel application-level differential checkpointing method for HPC applications with dynamic datasets, significantly reducing checkpointing time by up to 62% through a new fragmented dataset format and hash-based dirty data detection.

## Contribution

The work presents a new differential checkpointing implementation that supports dynamic dataset sizes using dataset fragmentation and hash algorithms, improving checkpoint efficiency.

## Key findings

- Achieved up to 62% reduction in checkpoint time.
- Demonstrated effectiveness on xPic, LULESH 2.0, and Heat2D applications.
- Supports dynamic-sized datasets with a novel dataset fragmentation approach.

## Abstract

High-performance computing (HPC) requires resilience techniques such as checkpointing in order to tolerate failures in supercomputers. As the number of nodes and memory in supercomputers keeps on increasing, the size of checkpoint data also increases dramatically, sometimes causing an I/O bottleneck. Differential checkpointing (dCP) aims to minimize the checkpointing overhead by only writing data differences. This is typically implemented at the memory page level, sometimes complemented with hashing algorithms. However, such a technique is unable to cope with dynamic-size datasets. In this work, we present a novel dCP implementation with a new file format that allows fragmentation of protected datasets in order to support dynamic sizes. We identify dirty data blocks using hash algorithms. In order to evaluate the dCP performance, we ported the HPC applications xPic, LULESH 2.0 and Heat2D and analyze them regarding their potential of reducing I/O with dCP and how this data reduction influences the checkpoint performance. In our experiments, we achieve reductions of up to 62% of the checkpoint time.