Dynamically Composing Domain-Data Selection with Clean-Data Selection by "Co-Curricular Learning" for Neural Machine Translation

TL;DR

This paper proposes a novel co-curricular learning approach that dynamically combines domain-data and clean-data selection for neural machine translation, improving transfer learning by explicitly modeling their interaction.

Contribution

It introduces a dynamic co-curricular learning method with an EM-style optimization to jointly select domain and clean data, addressing their interaction in data scheduling.

Findings

Effective in transfer learning across domains

Improves data scheduling for better translation quality

Demonstrates properties of co-curricular data scheduling

Abstract

Noise and domain are important aspects of data quality for neural machine translation. Existing research focus separately on domain-data selection, clean-data selection, or their static combination, leaving the dynamic interaction across them not explicitly examined. This paper introduces a "co-curricular learning" method to compose dynamic domain-data selection with dynamic clean-data selection, for transfer learning across both capabilities. We apply an EM-style optimization procedure to further refine the "co-curriculum". Experiment results and analysis with two domains demonstrate the effectiveness of the method and the properties of data scheduled by the co-curriculum.