ApproxIFER: A Model-Agnostic Approach to Resilient and Robust Prediction Serving Systems

TL;DR

ApproxIFER is a model-agnostic, resource-efficient approach for resilient prediction serving that handles multiple stragglers and Byzantine failures, significantly improving accuracy over existing parity-based methods.

Contribution

It introduces ApproxIFER, a novel, training-free method that is model-agnostic, scalable, and robust against adversarial workers in prediction serving systems.

Findings

Achieves up to 58% accuracy improvement over parity models.

Handles a general number of stragglers and Byzantine failures.

Scales better with the number of queries.

Abstract

Due to the surge of cloud-assisted AI services, the problem of designing resilient prediction serving systems that can effectively cope with stragglers/failures and minimize response delays has attracted much interest. The common approach for tackling this problem is replication which assigns the same prediction task to multiple workers. This approach, however, is very inefficient and incurs significant resource overheads. Hence, a learning-based approach known as parity model (ParM) has been recently proposed which learns models that can generate parities for a group of predictions in order to reconstruct the predictions of the slow/failed workers. While this learning-based approach is more resource-efficient than replication, it is tailored to the specific model hosted by the cloud and is particularly suitable for a small number of queries (typically less than four) and tolerating very few (mostly one) number of stragglers. Moreover, ParM does not handle Byzantine adversarial workers. We propose a different approach, named Approximate Coded Inference (ApproxIFER), that does not require training of any parity models, hence it is agnostic to the model hosted by the cloud and can be readily applied to different data domains and model architectures. Compared with earlier works, ApproxIFER can handle a general number of stragglers and scales significantly better with the number of queries. Furthermore, ApproxIFER is robust against Byzantine workers. Our extensive experiments on a large number of datasets and model architectures also show significant accuracy improvement by up to 58% over the parity model approaches.