Training Through Failure: Effects of Data Consistency in Parallel Machine Learning Training

TL;DR

This paper investigates how relaxing data consistency with a stateless parameter server during failures can improve training resilience and accuracy, offering a novel approach that maintains progress despite server downtime.

Contribution

The study introduces a novel stateless parameter server method that allows continued training during failures, outperforming traditional checkpointing and chain replication in resilience and accuracy.

Findings

Stateless parameter server maintains training progress during failures.

The approach improves accuracy by up to 10% despite stale updates.

Resource costs are comparable to standard checkpointing methods.

Abstract

In this study, we explore the impact of relaxing data consistency in parallel machine learning training during a failure using various parameter server configurations. Our failure recovery strategies include traditional checkpointing, chain replication (which ensures a backup server takes over in case of failure), and a novel stateless parameter server approach. In the stateless approach, workers continue generating gradient updates even if the parameter server is down, applying these updates once the server is back online. We compare these techniques to a standard checkpointing approach, where the training job is resumed from the latest checkpoint. To assess the resilience and performance of each configuration, we intentionally killed the parameter server during training for each experiment. Our experiment results indicate that the stateless parameter server approach continues to train towards convergence and improves accuracy as much as 10\% in the face of a failure despite using stale weights and gradients. The chain replication and checkpointing techniques demonstrate convergence but suffer from setbacks in accuracy due to restarting from old checkpoints. These results suggest that allowing workers to continue generating updates during server downtime and applying these updates later can effectively improve hardware utilization. Furthermore, despite higher resource usage, the stateless parameter server method incurs similar monetary costs in terms of hardware usage compared to standard checkpointing methods due to the pricing structure of common cloud providers.