Language Scaling for Universal Suggested Replies Model

TL;DR

This paper introduces a universal multilingual suggested replies model for Outlook that leverages multi-task continual learning with language adapters, achieving cross-lingual transfer, reduced costs, and improved user engagement.

Contribution

It presents a novel multi-task continual learning framework with language adapters to enable scalable, efficient, and high-quality multilingual suggested replies without joint training across regions.

Findings

Significant CTR improvement and characters saved in real user traffic.

65% reduction in training costs compared to per-language models.

Effective cross-lingual transfer with minimal catastrophic forgetting.

Abstract

We consider the problem of scaling automated suggested replies for Outlook email system to multiple languages. Faced with increased compute requirements and low resources for language expansion, we build a single universal model for improving the quality and reducing run-time costs of our production system. However, restricted data movement across regional centers prevents joint training across languages. To this end, we propose a multi-task continual learning framework, with auxiliary tasks and language adapters to learn universal language representation across regions. The experimental results show positive cross-lingual transfer across languages while reducing catastrophic forgetting across regions. Our online results on real user traffic show significant gains in CTR and characters saved, as well as 65% training cost reduction compared with per-language models. As a consequence, we have scaled the feature in multiple languages including low-resource markets.