Canoe : A System for Collaborative Learning for Neural Nets

TL;DR

Canoe is a system that enhances collaborative learning for neural networks in distributed environments by enabling efficient knowledge transfer and model adaptation, significantly reducing data transfer costs and improving performance.

Contribution

The paper introduces Canoe, a novel framework for knowledge transfer in neural networks that improves model adaptiveness and reduces data movement in distributed settings.

Findings

Knowledge transfer in Canoe improves model adaptiveness up to 3.5X.

Canoe reduces data movement costs by several orders of magnitude compared to federated learning.

Evaluation shows effectiveness across different neural network models in PyTorch and TensorFlow.

Abstract

For highly distributed environments such as edge computing, collaborative learning approaches eschew the dependence on a global, shared model, in favor of models tailored for each location. Creating tailored models for individual learning contexts reduces the amount of data transfer, while collaboration among peers provides acceptable model performance. Collaboration assumes, however, the availability of knowledge transfer mechanisms, which are not trivial for deep learning models where knowledge isn't easily attributed to precise model slices. We present Canoe - a framework that facilitates knowledge transfer for neural networks. Canoe provides new system support for dynamically extracting significant parameters from a helper node's neural network and uses this with a multi-model boosting-based approach to improve the predictive performance of the target node. The evaluation of Canoe with different PyTorch and TensorFlow neural network models demonstrates that the knowledge transfer mechanism improves the model's adaptiveness to changes up to 3.5X compared to learning in isolation, while affording several magnitudes reduction in data movement costs compared to federated learning.