Erasure Coded Neural Network Inference via Fisher Averaging

TL;DR

This paper introduces COIN, a novel erasure coding method for neural network inference that uses Fisher information to produce accurate, compute-efficient coded models, improving robustness against server failures.

Contribution

It formulates neural network coding as a KL barycenter problem and proposes COIN, a practical algorithm leveraging Fisher information for high-accuracy, efficient coded neural network outputs.

Findings

COIN outperforms baseline methods in accuracy of decoded outputs.

COIN is highly compute-efficient compared to existing approaches.

Experiments on real-world vision datasets validate the effectiveness of the method.

Abstract

Erasure-coded computing has been successfully used in cloud systems to reduce tail latency caused by factors such as straggling servers and heterogeneous traffic variations. A majority of cloud computing traffic now consists of inference on neural networks on shared resources where the response time of inference queries is also adversely affected by the same factors. However, current erasure coding techniques are largely focused on linear computations such as matrix-vector and matrix-matrix multiplications and hence do not work for the highly non-linear neural network functions. In this paper, we seek to design a method to code over neural networks, that is, given two or more neural network models, how to construct a coded model whose output is a linear combination of the outputs of the given neural networks. We formulate the problem as a KL barycenter problem and propose a practical algorithm COIN that leverages the diagonal Fisher information to create a coded model that approximately outputs the desired linear combination of outputs. We conduct experiments to perform erasure coding over neural networks trained on real-world vision datasets and show that the accuracy of the decoded outputs using COIN is significantly higher than other baselines while being extremely compute-efficient.