C-NMT: A Collaborative Inference Framework for Neural Machine Translation

TL;DR

This paper introduces C-NMT, a collaborative inference framework that reduces latency in neural machine translation by enabling edge-cloud cooperation, adapting existing CI methods to sequence generation tasks.

Contribution

It adapts collaborative inference techniques to neural machine translation, addressing sequence latency estimation and demonstrating significant latency reduction.

Findings

Latency reduced by up to 44%

Effective edge-cloud collaboration for NMT

Adapts CI methods to sequence tasks

Abstract

Collaborative Inference (CI) optimizes the latency and energy consumption of deep learning inference through the inter-operation of edge and cloud devices. Albeit beneficial for other tasks, CI has never been applied to the sequence- to-sequence mapping problem at the heart of Neural Machine Translation (NMT). In this work, we address the specific issues of collaborative NMT, such as estimating the latency required to generate the (unknown) output sequence, and show how existing CI methods can be adapted to these applications. Our experiments show that CI can reduce the latency of NMT by up to 44% compared to a non-collaborative approach.