Evaluating the Supervised and Zero-shot Performance of Multi-lingual Translation Models

TL;DR

This paper evaluates multilingual translation models' performance in both supervised and zero-shot settings across diverse language pairs, highlighting the impact of different decoder sharing strategies on translation quality.

Contribution

It provides the largest and most diverse evaluation of multilingual NMT models, comparing various decoder sharing methods for supervised and zero-shot translation.

Findings

Task-specific decoder parameters outperform fully shared decoders.

Zero-shot translation performance varies significantly with sharing strategies.

Largest evaluation to date with diverse zero-shot language pairs.

Abstract

We study several methods for full or partial sharing of the decoder parameters of multilingual NMT models. We evaluate both fully supervised and zero-shot translation performance in 110 unique translation directions using only the WMT 2019 shared task parallel datasets for training. We use additional test sets and re-purpose evaluation methods recently used for unsupervised MT in order to evaluate zero-shot translation performance for language pairs where no gold-standard parallel data is available. To our knowledge, this is the largest evaluation of multi-lingual translation yet conducted in terms of the total size of the training data we use, and in terms of the diversity of zero-shot translation pairs we evaluate. We conduct an in-depth evaluation of the translation performance of different models, highlighting the trade-offs between methods of sharing decoder parameters. We find that models which have task-specific decoder parameters outperform models where decoder parameters are fully shared across all tasks.

Tables

Table 1. Table 1: Training dataset statistics for our multilingual NMT experiments. # seen is the total number of segments seen during training. # available is the number of unique segments available in the parallel training datasets. # epochs is the number of passes made over the available training data – when this is < 1 absent 1 <1 , the available training data was only partially seen. % budget is the percentage of the training budget allocated to this pair of tasks.

		# seen	# available	# epochs	% budget
en-cs		3,466,692	51,136,198	0.06	10.7
en-de		2,678,808	3,054,632	0.88	8.3
en-fi		3,466,692	6,457,071	0.54	10.7
en-gu		1,260,615	137,905	9.14	3.9
en-kk		1,181,827	158,067	7.47	3.7
en-lt		3,624,269	2,283,272	1.59	11.2
en-ru		5,042,462	11,391,126	0.44	15.6
en-tr		1,575,769	207,678	7.58	4.9
en-zh		5,846,104	14,549,833	0.40	18.1
de-fr		4,097,000	1,980,332	2.06	12.7
Total		32,240,238	91,356,114	-	100

Table 2. Table 2: The WMT evaluation dataset used for each language pair.

		Evaluation Dataset
en-cs		newstest2018
en-de		newstest2018
en-fi		newstest2018
en-gu		newsdev2019
en-kk		newsdev2019
en-lt		newsdev2019
en-ru		newstest2018
en-tr		newstest2018
en-zh		newstest2018
de-fr		euelections_dev2019

Table 3. Table 3: Overall results for supervised and zero-shot tasks. Tokenized BLEU scores are computed by concatenating all of the hypotheses for all translation directions, and computing BLEU with respect to the concatenated references. We use the sentencepiece-segmented hypotheses and references to avoid issues with tokenization of multi-lingual hypotheses and references.

		Prepend	Emb	Dec	Attn
Supervised		$23.382$	$23.405$	$23.963$	$24.071$
Zero-Shot-TED		$10.606$	$7.790$	$12.601$	$12.364$
Zero-Shot-Pivot		$16.866$	$18.061$	$13.952$	$15.143$
Zero-Shot-Parallel-Pivot		$12.646$	$11.256$	$12.277$	$13.082$

Table 4. Table 4: Results for all task pairs in the WMT 2019 news-translation shared task where parallel training data is available.

		Prepend	Emb	Dec	Attn
cs-en		$20.192$	$20.234$	$20.945$	$20.894$
en-cs		$12.365$	$12.703$	$13.680$	$13.304$
de-en		$26.204$	$26.064$	$27.430$	$27.078$
en-de		$23.201$	$23.406$	$25.706$	$25.240$
fi-en		$13.713$	$13.542$	$14.355$	$14.159$
en-fi		$8.316$	$8.049$	$9.354$	$9.173$
gu-en		$15.43$	$15.439$	$15.666$	$15.403$
en-gu		$8.098$	$7.770$	$5.107$	$7.256$
kk-en		$14.42$	$13.985$	$14.336$	$13.880$
en-kk		$5.557$	$5.197$	$1.851$	$4.614$
lt-en		$18.550$	$18.865$	$19.345$	$19.047$
en-lt		$12.754$	$13.022$	$14.406$	$13.726$
ru-en		$20.752$	$20.573$	$21.323$	$21.344$
en-ru		$15.538$	$15.930$	$17.029$	$16.735$
tr-en		$14.81$	$14.953$	$15.188$	$15.074$
en-tr		$10.318$	$10.034$	$10.931$	$11.332$
zh-en		$13.53$	$13.705$	$14.069$	$13.727$
en-zh		$24.2$	$24.4$	$25.6$	$25.4$
fr-de		$18.553$	$18.435$	$19.89$	$19.294$
de-fr		$21.170$	$22.114$	$21.730$	$22.566$

Table 5. Table 5: Zero-shot translation results for ru → → \rightarrow * → → \rightarrow ru Note that BLEU scores are computed by translating SRC → Pivot → \savestack \tmpbox \stretchto \scaleto \scalerel *[ S ​ R ​ C ]^ 2.4ex \stackon [-6.9pt]SRC \tmpbox → SRC Pivot → \savestack \tmpbox \stretchto \scaleto \scalerel *[ S ⁢ R ⁢ C ]^ 2.4ex \stackon [-6.9pt]SRC \tmpbox \textsc{SRC}\rightarrow\textsc{Pivot}\rightarrow\textsc{\savestack{\tmpbox}{\stretchto{\scaleto{\scalerel*[\widthof{$SRC$}]{\kern 0.1pt\mathchar 866\relax\kern 0.1pt}{\rule{0.0pt}{702.78297pt}}}{}}{2.4ex}}\stackon[-6.9pt]{SRC}{\tmpbox}} , and computing the score between SRC and \savestack \tmpbox \stretchto \scaleto \scalerel *[ S ​ R ​ C 𝑆 𝑅 𝐶 SRC ]^ 2.4ex \stackon [-6.9pt]SRC \tmpbox . Systems which do not pass the language identification filter are struck-through and removed from global evaluation. Note that parallel training data was available for ru → → \rightarrow en .

		Prepend	Emb	Dec	Attn
ru$\to$cs$\to$ru		$20.867$	$23.843$	$20.814$	$20.963$
ru$\to$de$\to$ru		$14.616$	$11.872$	$16.365$	$15.590$
ru$\to$en$\to$ru*		$21.674$	$22.224$	$23.937$	$23.180$
ru$\to$fi$\to$ru		$11.161$	$17.001$	$12.059$	$11.589$
ru$\to$fr$\to$ru		$13.755$	$15.351$	$14.098$	$15.149$
ru$\to$gu$\to$ru		$10.3$	$9.6$	$3.5$	$5.1$
ru$\to$kk$\to$ru		$5.766$	$19.565$	$1.030$	$2.212$
ru$\to$lt$\to$ru		$16.880$	$22.037$	$16.536$	$16.639$
ru$\to$tr$\to$ru		$7.914$	$10.232$	$7.447$	$7.661$
ru$\to$zh$\to$ru		$8.843$	$10.474$	$9.099$	$8.483$

Table 6. Table 6: Out of 110 pivot translation tasks, how many failed the language identification check?

		Prepend	Emb	Dec	Attn
# Failed Pivot Tasks		3	31	1	1

Table 7. Table 7: Zero-Shot Pivot translation results in all directions, for all models. Rows indicate source language, columns indicate pivot language. For example, cell ( 1 , 2 ) 1 2 (1,2) contains the results for cs → → \rightarrow de → → \rightarrow cs . Runs which did not pass the language identification filter are struck-through. The MT-matrix ( http://matrix.statmt.org/matrix ) was the inspiration for this rendering. Note this table shows results for all translation directions, but we only use the zero-shot pairs for evaluation.

Prepend	cs	$15.72$	$19.64$	$11.36$	$11.10$	$8.67$	$3.55$	$16.57$	$17.41$	$7.75$	$7.52$
Emb		$9.30$	$19.91$	$24.36$	$12.39$	$9.28$	$3.32$	$28.41$	$15.80$	$10.16$	$7.53$
Dec		$17.72$	$21.26$	$11.45$	$13.32$	$3.26$	$0.74$	$13.95$	$17.83$	$6.72$	$7.51$
Attn		$17.45$	$21.55$	$11.60$	$13.81$	$4.49$	$1.84$	$14.36$	$17.42$	$7.21$	$7.60$
Prepend	$22.25$	de	$27.05$	$16.37$	$25.32$	$11.20$	$5.32$	$18.29$	$18.13$	$12.39$	$12.34$
Emb	$41.51$		$27.62$	$38.00$	$25.08$	$19.42$	$8.47$	$40.67$	$23.57$	$29.89$	$19.62$
Dec	$23.35$		$29.81$	$15.64$	$25.84$	$3.99$	$0.93$	$16.22$	$19.14$	$11.23$	$12.19$
Attn	$22.79$		$29.02$	$15.91$	$26.33$	$6.40$	$2.74$	$17.05$	$17.99$	$11.19$	$12.06$
Prepend	$35.41$	$37.05$	en	$24.62$	$34.46$	$22.13$	$9.26$	$29.18$	$32.94$	$23.29$	$25.61$
Emb	$36.53$	$37.44$		$25.89$	$35.00$	$21.49$	$9.26$	$30.37$	$33.85$	$24.24$	$26.57$
Dec	$35.82$	$37.49$		$25.76$	$32.62$	$10.11$	$1.58$	$29.71$	$33.23$	$22.76$	$26.34$
Attn	$36.89$	$36.60$		$25.88$	$34.37$	$15.66$	$6.20$	$30.33$	$33.86$	$23.63$	$26.80$
Prepend	$12.05$	$10.98$	$14.33$	fi	$7.15$	$5.62$	$2.55$	$13.10$	$9.16$	$6.59$	$6.24$
Emb	$19.86$	$7.87$	$14.65$		$8.17$	$6.20$	$4.22$	$23.76$	$11.14$	$12.00$	$6.76$
Dec	$11.23$	$11.69$	$15.41$		$9.77$	$3.02$	$0.54$	$10.74$	$9.85$	$6.15$	$5.78$
Attn	$12.15$	$11.48$	$15.01$		$10.00$	$4.18$	$1.70$	$10.87$	$9.77$	$6.47$	$5.79$
Prepend	$25.56$	$32.65$	$31.87$	$17.83$	fr	$17.10$	$7.79$	$20.47$	$22.88$	$16.04$	$15.27$
Emb	$25.99$	$32.02$	$36.53$	$20.42$		$12.41$	$5.02$	$24.53$	$22.49$	$15.58$	$14.10$
Dec	$25.68$	$32.90$	$33.91$	$18.39$		$5.86$	$1.51$	$20.24$	$23.61$	$14.44$	$15.07$
Attn	$26.04$	$33.16$	$34.34$	$19.54$		$8.51$	$4.54$	$20.99$	$24.56$	$15.49$	$15.36$
Prepend	$5.14$	$5.72$	$8.17$	$4.17$	$3.98$	gu	$2.01$	$5.29$	$5.20$	$3.50$	$3.94$
Emb	$5.30$	$4.64$	$7.62$	$5.39$	$4.37$		$1.55$	$6.22$	$3.60$	$4.20$	$3.67$
Dec	$3.54$	$3.50$	$5.48$	$3.24$	$3.23$		$0.51$	$3.56$	$3.39$	$2.86$	$2.65$
Attn	$4.88$	$5.05$	$7.56$	$4.43$	$4.52$		$1.04$	$4.60$	$4.53$	$4.04$	$4.00$
Prepend	$4.86$	$4.77$	$7.05$	$3.40$	$2.83$	$4.29$	kk	$4.61$	$4.90$	$4.43$	$3.88$
Emb	$3.36$	$3.40$	$6.36$	$3.82$	$3.50$	$1.87$		$3.93$	$3.62$	$4.48$	$2.40$
Dec	$1.65$	$1.76$	$2.18$	$1.55$	$1.47$	$1.07$		$1.53$	$1.65$	$1.74$	$1.19$
Attn	$3.89$	$3.96$	$5.34$	$3.22$	$3.19$	$2.26$		$3.61$	$3.89$	$4.31$	$3.19$
Prepend	$18.84$	$14.48$	$17.92$	$13.81$	$10.37$	$9.88$	$4.67$	lt	$16.82$	$9.99$	$8.98$
Emb	$30.52$	$11.99$	$18.33$	$30.70$	$10.82$	$10.60$	$6.75$		$18.21$	$19.71$	$13.71$
Dec	$16.74$	$13.56$	$18.87$	$12.80$	$11.60$	$3.66$	$0.93$		$16.10$	$8.03$	$8.54$
Attn	$16.95$	$13.86$	$18.75$	$12.54$	$12.36$	$5.46$	$1.89$		$15.70$	$8.72$	$9.16$
Prepend	$20.87$	$14.62$	$21.67$	$11.16$	$13.76$	$10.26$	$5.77$	$16.88$	ru	$7.91$	$8.84$
Emb	$23.84$	$11.87$	$22.22$	$17.00$	$15.35$	$9.56$	$19.57$	$22.04$		$10.23$	$10.47$
Dec	$20.81$	$16.37$	$23.94$	$12.06$	$14.10$	$3.54$	$1.03$	$16.54$		$7.45$	$9.10$
Attn	$20.96$	$15.59$	$23.18$	$11.59$	$15.15$	$5.05$	$2.21$	$16.64$		$7.66$	$8.48$
Prepend	$9.11$	$8.17$	$13.33$	$7.42$	$7.35$	$8.77$	$5.60$	$8.86$	$7.11$	tr	$6.40$
Emb	$12.66$	$7.19$	$12.64$	$14.58$	$7.48$	$6.65$	$4.10$	$17.28$	$6.62$		$6.36$
Dec	$7.15$	$7.62$	$13.05$	$6.47$	$6.83$	$2.50$	$0.65$	$6.44$	$5.63$		$5.28$
Attn	$7.33$	$8.13$	$13.41$	$6.63$	$7.27$	$3.92$	$1.89$	$6.73$	$5.56$		$5.26$
Prepend	$20.4$	$19.6$	$29.0$	$17.1$	$17.4$	$18.2$	$8.4$	$20.2$	$19.5$	$17.4$	zh
Emb	$20.1$	$16.9$	$29.4$	$19.6$	$17.8$	$11.9$	$6.6$	$22.8$	$18.3$	$16.7$
Dec	$19.2$	$19.4$	$30.2$	$16.6$	$17.6$	$7.2$	$2.2$	$19.5$	$20.1$	$16.3$
Attn	$19.8$	$20.4$	$30.0$	$16.7$	$18.2$	$11.0$	$5.0$	$18.6$	$19.4$	$17.1$

Table 8. Table 8: Zero-Shot Parallel Pivot translation results in all directions, for all models. Rows indicate source language, columns indicate pivot language. Note en is omitted from this table because it is used as source language when translating into the pivot language for all target tasks except fr .

Prepend	cs	$\mathrm{14.414\hspace{0.17em}874\hspace{0.17em}476\hspace{0.17em}078\hspace{0.17em}497}$	$\mathrm{6.088\hspace{0.17em}300\hspace{0.17em}056\hspace{0.17em}286\hspace{0.17em}523}$	$\mathrm{14.935\hspace{0.17em}391\hspace{0.17em}794\hspace{0.17em}352\hspace{0.17em}618}$	$\mathrm{3.635\hspace{0.17em}169\hspace{0.17em}955\hspace{0.17em}693\hspace{0.17em}632}$	$\mathrm{3.658\hspace{0.17em}600\hspace{0.17em}207\hspace{0.17em}726\hspace{0.17em}975}$	$\mathrm{9.421\hspace{0.17em}033\hspace{0.17em}501\hspace{0.17em}890\hspace{0.17em}738}$	$\mathrm{10.471\hspace{0.17em}132\hspace{0.17em}066\hspace{0.17em}757\hspace{0.17em}772}$	$\mathrm{6.399\hspace{0.17em}931\hspace{0.17em}994\hspace{0.17em}955\hspace{0.17em}041\hspace{0.17em}5}$	$18.1$
Emb		$\mathrm{11.520\hspace{0.17em}446\hspace{0.17em}690\hspace{0.17em}250\hspace{0.17em}273}$	$\mathrm{3.737\hspace{0.17em}494\hspace{0.17em}405\hspace{0.17em}543\hspace{0.17em}1}$	$\mathrm{17.338\hspace{0.17em}392\hspace{0.17em}140\hspace{0.17em}179\hspace{0.17em}163}$	$\mathrm{2.627\hspace{0.17em}873\hspace{0.17em}859\hspace{0.17em}060\hspace{0.17em}852\hspace{0.17em}4}$	$\mathrm{2.575\hspace{0.17em}698\hspace{0.17em}772\hspace{0.17em}110\hspace{0.17em}214\hspace{0.17em}5}$	$\mathrm{6.172\hspace{0.17em}204\hspace{0.17em}545\hspace{0.17em}169\hspace{0.17em}574}$	$\mathrm{9.076\hspace{0.17em}473\hspace{0.17em}467\hspace{0.17em}923\hspace{0.17em}657}$	$\mathrm{4.219\hspace{0.17em}306\hspace{0.17em}814\hspace{0.17em}316\hspace{0.17em}005}$	$14.3$
Dec		$\mathrm{15.632\hspace{0.17em}544\hspace{0.17em}171\hspace{0.17em}663\hspace{0.17em}184}$	$\mathrm{6.396\hspace{0.17em}894\hspace{0.17em}556\hspace{0.17em}548\hspace{0.17em}482\hspace{0.17em}5}$	$\mathrm{15.454\hspace{0.17em}741\hspace{0.17em}952\hspace{0.17em}185\hspace{0.17em}957}$	$\mathrm{3.109\hspace{0.17em}018\hspace{0.17em}136\hspace{0.17em}291\hspace{0.17em}276\hspace{0.17em}8}$	$\mathrm{1.608\hspace{0.17em}318\hspace{0.17em}724\hspace{0.17em}359\hspace{0.17em}964\hspace{0.17em}4}$	$\mathrm{10.234\hspace{0.17em}462\hspace{0.17em}099\hspace{0.17em}948\hspace{0.17em}114}$	$\mathrm{11.762\hspace{0.17em}473\hspace{0.17em}035\hspace{0.17em}584\hspace{0.17em}188}$	$\mathrm{6.860\hspace{0.17em}612\hspace{0.17em}723\hspace{0.17em}941\hspace{0.17em}004}$	$19.7$
Attn		$\mathrm{16.114\hspace{0.17em}497\hspace{0.17em}912\hspace{0.17em}009\hspace{0.17em}874}$	$\mathrm{6.555\hspace{0.17em}280\hspace{0.17em}926\hspace{0.17em}852\hspace{0.17em}87}$	$\mathrm{15.348\hspace{0.17em}641\hspace{0.17em}271\hspace{0.17em}057\hspace{0.17em}106}$	$\mathrm{4.298\hspace{0.17em}577\hspace{0.17em}657\hspace{0.17em}975\hspace{0.17em}255\hspace{0.17em}5}$	$\mathrm{3.262\hspace{0.17em}624\hspace{0.17em}692\hspace{0.17em}908\hspace{0.17em}609\hspace{0.17em}3}$	$\mathrm{9.743\hspace{0.17em}903\hspace{0.17em}318\hspace{0.17em}544\hspace{0.17em}128}$	$\mathrm{11.572\hspace{0.17em}168\hspace{0.17em}996\hspace{0.17em}493\hspace{0.17em}767}$	$\mathrm{7.124\hspace{0.17em}717\hspace{0.17em}894\hspace{0.17em}702\hspace{0.17em}883}$	$20.1$
Prepend	$\mathrm{8.529\hspace{0.17em}012\hspace{0.17em}978\hspace{0.17em}111\hspace{0.17em}979}$	de	$\mathrm{6.204\hspace{0.17em}265\hspace{0.17em}103\hspace{0.17em}992\hspace{0.17em}004}$		$\mathrm{4.615\hspace{0.17em}216\hspace{0.17em}803\hspace{0.17em}701\hspace{0.17em}71}$	$\mathrm{3.846\hspace{0.17em}043\hspace{0.17em}539\hspace{0.17em}994\hspace{0.17em}992\hspace{0.17em}6}$	$\mathrm{8.588\hspace{0.17em}704\hspace{0.17em}000\hspace{0.17em}658\hspace{0.17em}291}$	$\mathrm{9.837\hspace{0.17em}263\hspace{0.17em}426\hspace{0.17em}544\hspace{0.17em}263}$	$\mathrm{6.361\hspace{0.17em}128\hspace{0.17em}962\hspace{0.17em}887\hspace{0.17em}625\hspace{0.17em}5}$	$18.4$
Emb	$\mathrm{3.899\hspace{0.17em}521\hspace{0.17em}793\hspace{0.17em}415\hspace{0.17em}068\hspace{0.17em}3}$		$\mathrm{3.280\hspace{0.17em}771\hspace{0.17em}304\hspace{0.17em}858\hspace{0.17em}457\hspace{0.17em}7}$		$\mathrm{2.518\hspace{0.17em}140\hspace{0.17em}793\hspace{0.17em}731\hspace{0.17em}33}$	$\mathrm{2.535\hspace{0.17em}293\hspace{0.17em}037\hspace{0.17em}387\hspace{0.17em}257}$	$\mathrm{4.651\hspace{0.17em}232\hspace{0.17em}615\hspace{0.17em}675\hspace{0.17em}921}$	$\mathrm{7.107\hspace{0.17em}404\hspace{0.17em}524\hspace{0.17em}144\hspace{0.17em}222}$	$\mathrm{3.632\hspace{0.17em}683\hspace{0.17em}733\hspace{0.17em}352\hspace{0.17em}523}$	$14.4$
Dec	$\mathrm{9.975\hspace{0.17em}648\hspace{0.17em}609\hspace{0.17em}496\hspace{0.17em}43}$		$\mathrm{7.272\hspace{0.17em}112\hspace{0.17em}396\hspace{0.17em}298\hspace{0.17em}628}$		$\mathrm{3.323\hspace{0.17em}701\hspace{0.17em}739\hspace{0.17em}073\hspace{0.17em}321\hspace{0.17em}4}$	$\mathrm{1.540\hspace{0.17em}998\hspace{0.17em}769\hspace{0.17em}276\hspace{0.17em}214\hspace{0.17em}8}$	$\mathrm{9.118\hspace{0.17em}239\hspace{0.17em}802\hspace{0.17em}147\hspace{0.17em}878}$	$\mathrm{10.902\hspace{0.17em}695\hspace{0.17em}232\hspace{0.17em}958\hspace{0.17em}196}$	$\mathrm{7.357\hspace{0.17em}109\hspace{0.17em}338\hspace{0.17em}928\hspace{0.17em}602}$	$20.4$
Attn	$\mathrm{9.817\hspace{0.17em}147\hspace{0.17em}870\hspace{0.17em}321\hspace{0.17em}51}$		$\mathrm{7.309\hspace{0.17em}903\hspace{0.17em}125\hspace{0.17em}052\hspace{0.17em}365}$		$\mathrm{4.462\hspace{0.17em}257\hspace{0.17em}571\hspace{0.17em}072\hspace{0.17em}914}$	$\mathrm{3.343\hspace{0.17em}272\hspace{0.17em}829\hspace{0.17em}533\hspace{0.17em}058}$	$\mathrm{9.174\hspace{0.17em}772\hspace{0.17em}509\hspace{0.17em}031\hspace{0.17em}467}$	$\mathrm{10.874\hspace{0.17em}952\hspace{0.17em}625\hspace{0.17em}972\hspace{0.17em}854}$	$\mathrm{7.460\hspace{0.17em}488\hspace{0.17em}213\hspace{0.17em}823\hspace{0.17em}105}$	$20.6$
Prepend	$\mathrm{6.603\hspace{0.17em}005\hspace{0.17em}125\hspace{0.17em}045\hspace{0.17em}748}$	$\mathrm{10.535\hspace{0.17em}185\hspace{0.17em}507\hspace{0.17em}899\hspace{0.17em}82}$	fi	$\mathrm{11.483\hspace{0.17em}609\hspace{0.17em}152\hspace{0.17em}547\hspace{0.17em}563}$	$\mathrm{3.393\hspace{0.17em}383\hspace{0.17em}584\hspace{0.17em}953\hspace{0.17em}865\hspace{0.17em}4}$	$\mathrm{2.794\hspace{0.17em}672\hspace{0.17em}392\hspace{0.17em}215\hspace{0.17em}475\hspace{0.17em}5}$	$\mathrm{7.662\hspace{0.17em}530\hspace{0.17em}326\hspace{0.17em}333\hspace{0.17em}799}$	$\mathrm{7.572\hspace{0.17em}185\hspace{0.17em}485\hspace{0.17em}460\hspace{0.17em}277}$	$\mathrm{5.121\hspace{0.17em}476\hspace{0.17em}903\hspace{0.17em}438\hspace{0.17em}894}$	$15.0$
Emb	$\mathrm{4.041\hspace{0.17em}685\hspace{0.17em}420\hspace{0.17em}814\hspace{0.17em}64}$	$\mathrm{8.629\hspace{0.17em}108\hspace{0.17em}904\hspace{0.17em}386\hspace{0.17em}111}$		$\mathrm{16.277\hspace{0.17em}060\hspace{0.17em}931\hspace{0.17em}629\hspace{0.17em}244}$	$\mathrm{2.664\hspace{0.17em}907\hspace{0.17em}617\hspace{0.17em}310\hspace{0.17em}452}$	$\mathrm{1.786\hspace{0.17em}875\hspace{0.17em}483\hspace{0.17em}961\hspace{0.17em}143\hspace{0.17em}2}$	$\mathrm{5.689\hspace{0.17em}423\hspace{0.17em}928\hspace{0.17em}328\hspace{0.17em}953}$	$\mathrm{5.794\hspace{0.17em}187\hspace{0.17em}594\hspace{0.17em}636\hspace{0.17em}456\hspace{0.17em}5}$	$\mathrm{3.436\hspace{0.17em}169\hspace{0.17em}811\hspace{0.17em}233\hspace{0.17em}835\hspace{0.17em}7}$	$12.5$
Dec	$\mathrm{7.543\hspace{0.17em}628\hspace{0.17em}517\hspace{0.17em}658\hspace{0.17em}198}$	$\mathrm{12.323\hspace{0.17em}511\hspace{0.17em}964\hspace{0.17em}164\hspace{0.17em}32}$		$\mathrm{12.024\hspace{0.17em}796\hspace{0.17em}538\hspace{0.17em}940\hspace{0.17em}151}$	$\mathrm{3.136\hspace{0.17em}648\hspace{0.17em}302\hspace{0.17em}610\hspace{0.17em}435}$	$\mathrm{1.462\hspace{0.17em}054\hspace{0.17em}322\hspace{0.17em}962\hspace{0.17em}005\hspace{0.17em}3}$	$\mathrm{8.319\hspace{0.17em}030\hspace{0.17em}530\hspace{0.17em}455\hspace{0.17em}747}$	$\mathrm{8.542\hspace{0.17em}335\hspace{0.17em}706\hspace{0.17em}790\hspace{0.17em}11}$	$\mathrm{5.398\hspace{0.17em}713\hspace{0.17em}804\hspace{0.17em}085\hspace{0.17em}403}$	$16.8$
Attn	$\mathrm{7.440\hspace{0.17em}593\hspace{0.17em}966\hspace{0.17em}391\hspace{0.17em}604}$	$\mathrm{12.102\hspace{0.17em}395\hspace{0.17em}510\hspace{0.17em}828\hspace{0.17em}641}$		$\mathrm{12.345\hspace{0.17em}944\hspace{0.17em}260\hspace{0.17em}775\hspace{0.17em}198}$	$\mathrm{3.809\hspace{0.17em}483\hspace{0.17em}596\hspace{0.17em}665\hspace{0.17em}484\hspace{0.17em}3}$	$\mathrm{2.915\hspace{0.17em}752\hspace{0.17em}521\hspace{0.17em}723\hspace{0.17em}301}$	$\mathrm{8.109\hspace{0.17em}703\hspace{0.17em}433\hspace{0.17em}327\hspace{0.17em}54}$	$\mathrm{7.959\hspace{0.17em}902\hspace{0.17em}373\hspace{0.17em}162\hspace{0.17em}625}$	$\mathrm{5.457\hspace{0.17em}070\hspace{0.17em}325\hspace{0.17em}591\hspace{0.17em}112}$	$16.7$
Prepend		$\mathrm{16.492\hspace{0.17em}407\hspace{0.17em}179\hspace{0.17em}331\hspace{0.17em}877}$	$\mathrm{4.752\hspace{0.17em}291\hspace{0.17em}842\hspace{0.17em}207\hspace{0.17em}557}$	fr	$\mathrm{3.471\hspace{0.17em}726\hspace{0.17em}686\hspace{0.17em}742\hspace{0.17em}642}$	$\mathrm{2.824\hspace{0.17em}734\hspace{0.17em}353\hspace{0.17em}118\hspace{0.17em}774}$	$\mathrm{7.372\hspace{0.17em}021\hspace{0.17em}095\hspace{0.17em}079\hspace{0.17em}622}$	$\mathrm{9.251\hspace{0.17em}598\hspace{0.17em}700\hspace{0.17em}545\hspace{0.17em}761}$	$\mathrm{5.976\hspace{0.17em}424\hspace{0.17em}219\hspace{0.17em}333\hspace{0.17em}110\hspace{0.17em}5}$	$17.2$
Emb		$\mathrm{17.053\hspace{0.17em}799\hspace{0.17em}386\hspace{0.17em}199\hspace{0.17em}714}$	$\mathrm{5.266\hspace{0.17em}444\hspace{0.17em}633\hspace{0.17em}250\hspace{0.17em}403}$		$\mathrm{3.947\hspace{0.17em}862\hspace{0.17em}692\hspace{0.17em}303\hspace{0.17em}706\hspace{0.17em}6}$	$\mathrm{3.109\hspace{0.17em}710\hspace{0.17em}878\hspace{0.17em}219\hspace{0.17em}086\hspace{0.17em}5}$	$\mathrm{8.319\hspace{0.17em}065\hspace{0.17em}383\hspace{0.17em}772\hspace{0.17em}09}$	$\mathrm{10.971\hspace{0.17em}688\hspace{0.17em}625\hspace{0.17em}246\hspace{0.17em}923}$	$\mathrm{6.814\hspace{0.17em}107\hspace{0.17em}473\hspace{0.17em}702\hspace{0.17em}56}$	$18.5$
Dec		$\mathrm{16.121\hspace{0.17em}694\hspace{0.17em}982\hspace{0.17em}857\hspace{0.17em}697}$	$\mathrm{6.347\hspace{0.17em}529\hspace{0.17em}574\hspace{0.17em}193\hspace{0.17em}385}$		$\mathrm{2.863\hspace{0.17em}460\hspace{0.17em}859\hspace{0.17em}945\hspace{0.17em}674\hspace{0.17em}6}$	$\mathrm{1.422\hspace{0.17em}711\hspace{0.17em}061\hspace{0.17em}890\hspace{0.17em}189\hspace{0.17em}5}$	$\mathrm{8.063\hspace{0.17em}492\hspace{0.17em}046\hspace{0.17em}977\hspace{0.17em}99}$	$\mathrm{10.590\hspace{0.17em}159\hspace{0.17em}179\hspace{0.17em}325\hspace{0.17em}584}$	$\mathrm{6.505\hspace{0.17em}893\hspace{0.17em}156\hspace{0.17em}035\hspace{0.17em}486}$	$18.7$
Attn		$\mathrm{16.330\hspace{0.17em}666\hspace{0.17em}025\hspace{0.17em}143\hspace{0.17em}163}$	$\mathrm{6.236\hspace{0.17em}562\hspace{0.17em}303\hspace{0.17em}120\hspace{0.17em}987}$		$\mathrm{4.877\hspace{0.17em}199\hspace{0.17em}895\hspace{0.17em}522\hspace{0.17em}516}$	$\mathrm{2.972\hspace{0.17em}976\hspace{0.17em}714\hspace{0.17em}993\hspace{0.17em}755}$	$\mathrm{8.511\hspace{0.17em}123\hspace{0.17em}673\hspace{0.17em}596\hspace{0.17em}467}$	$\mathrm{10.544\hspace{0.17em}058\hspace{0.17em}867\hspace{0.17em}953\hspace{0.17em}233}$	$\mathrm{6.795\hspace{0.17em}706\hspace{0.17em}716\hspace{0.17em}422\hspace{0.17em}605}$	$19.4$
Prepend	$\mathrm{4.233\hspace{0.17em}605\hspace{0.17em}321\hspace{0.17em}313\hspace{0.17em}369}$	$\mathrm{6.926\hspace{0.17em}930\hspace{0.17em}321\hspace{0.17em}812\hspace{0.17em}668}$	$\mathrm{3.167\hspace{0.17em}242\hspace{0.17em}494\hspace{0.17em}969\hspace{0.17em}903\hspace{0.17em}6}$	$\mathrm{7.889\hspace{0.17em}926\hspace{0.17em}128\hspace{0.17em}405\hspace{0.17em}102\hspace{0.17em}5}$	gu	$\mathrm{2.409\hspace{0.17em}042\hspace{0.17em}021\hspace{0.17em}353\hspace{0.17em}989\hspace{0.17em}6}$	$\mathrm{5.090\hspace{0.17em}794\hspace{0.17em}046\hspace{0.17em}286\hspace{0.17em}559}$	$\mathrm{5.455\hspace{0.17em}129\hspace{0.17em}103\hspace{0.17em}849}$	$\mathrm{3.357\hspace{0.17em}518\hspace{0.17em}383\hspace{0.17em}164\hspace{0.17em}683\hspace{0.17em}3}$	$11.5$
Emb	$\mathrm{2.494\hspace{0.17em}471\hspace{0.17em}458\hspace{0.17em}066\hspace{0.17em}196\hspace{0.17em}4}$	$\mathrm{5.561\hspace{0.17em}736\hspace{0.17em}971\hspace{0.17em}923\hspace{0.17em}027}$	$\mathrm{2.271\hspace{0.17em}381\hspace{0.17em}517\hspace{0.17em}971\hspace{0.17em}598\hspace{0.17em}4}$	$\mathrm{9.026\hspace{0.17em}466\hspace{0.17em}247\hspace{0.17em}162\hspace{0.17em}249}$		$\mathrm{1.873\hspace{0.17em}182\hspace{0.17em}427\hspace{0.17em}095\hspace{0.17em}073\hspace{0.17em}4}$	$\mathrm{3.472\hspace{0.17em}331\hspace{0.17em}320\hspace{0.17em}266\hspace{0.17em}484\hspace{0.17em}5}$	$\mathrm{3.522\hspace{0.17em}197\hspace{0.17em}450\hspace{0.17em}529\hspace{0.17em}081}$	$\mathrm{2.432\hspace{0.17em}400\hspace{0.17em}632\hspace{0.17em}320\hspace{0.17em}57}$	$8.7$
Dec	$\mathrm{2.430\hspace{0.17em}849\hspace{0.17em}028\hspace{0.17em}458\hspace{0.17em}058\hspace{0.17em}4}$	$\mathrm{3.962\hspace{0.17em}138\hspace{0.17em}626\hspace{0.17em}272\hspace{0.17em}385\hspace{0.17em}4}$	$\mathrm{2.010\hspace{0.17em}949\hspace{0.17em}157\hspace{0.17em}391\hspace{0.17em}496\hspace{0.17em}4}$	$\mathrm{3.975\hspace{0.17em}855\hspace{0.17em}379\hspace{0.17em}856\hspace{0.17em}164}$		$\mathrm{1.034\hspace{0.17em}894\hspace{0.17em}734\hspace{0.17em}846\hspace{0.17em}992\hspace{0.17em}1}$	$\mathrm{3.085\hspace{0.17em}957\hspace{0.17em}948\hspace{0.17em}753\hspace{0.17em}362\hspace{0.17em}6}$	$\mathrm{3.106\hspace{0.17em}690\hspace{0.17em}936\hspace{0.17em}579\hspace{0.17em}163}$	$\mathrm{2.227\hspace{0.17em}729\hspace{0.17em}757\hspace{0.17em}703\hspace{0.17em}559\hspace{0.17em}5}$	$7.6$
Attn	$\mathrm{3.652\hspace{0.17em}661\hspace{0.17em}838\hspace{0.17em}021\hspace{0.17em}046\hspace{0.17em}5}$	$\mathrm{5.726\hspace{0.17em}423\hspace{0.17em}710\hspace{0.17em}912\hspace{0.17em}327}$	$\mathrm{2.894\hspace{0.17em}265\hspace{0.17em}984\hspace{0.17em}875\hspace{0.17em}042\hspace{0.17em}3}$	$\mathrm{5.798\hspace{0.17em}558\hspace{0.17em}288\hspace{0.17em}076\hspace{0.17em}469}$		$\mathrm{2.055\hspace{0.17em}470\hspace{0.17em}040\hspace{0.17em}518\hspace{0.17em}191\hspace{0.17em}6}$	$\mathrm{4.283\hspace{0.17em}554\hspace{0.17em}608\hspace{0.17em}187\hspace{0.17em}96}$	$\mathrm{4.058\hspace{0.17em}425\hspace{0.17em}462\hspace{0.17em}342\hspace{0.17em}932}$	$\mathrm{3.052\hspace{0.17em}871\hspace{0.17em}140\hspace{0.17em}936\hspace{0.17em}088\hspace{0.17em}2}$	$10.5$
Prepend	$\mathrm{2.082\hspace{0.17em}328\hspace{0.17em}088\hspace{0.17em}855\hspace{0.17em}985}$	$\mathrm{2.406\hspace{0.17em}833\hspace{0.17em}527\hspace{0.17em}425\hspace{0.17em}930\hspace{0.17em}3}$	$\mathrm{1.668\hspace{0.17em}480\hspace{0.17em}846\hspace{0.17em}518\hspace{0.17em}301\hspace{0.17em}2}$	$\mathrm{4.878\hspace{0.17em}016\hspace{0.17em}228\hspace{0.17em}117\hspace{0.17em}733}$	$\mathrm{1.304\hspace{0.17em}187\hspace{0.17em}975\hspace{0.17em}382\hspace{0.17em}530\hspace{0.17em}5}$	kk	$\mathrm{2.947\hspace{0.17em}119\hspace{0.17em}415\hspace{0.17em}588\hspace{0.17em}47}$	$\mathrm{2.726\hspace{0.17em}690\hspace{0.17em}632\hspace{0.17em}145\hspace{0.17em}856}$	$\mathrm{2.122\hspace{0.17em}602\hspace{0.17em}396\hspace{0.17em}039\hspace{0.17em}762}$	$7.9$
Emb	$\mathrm{1.176\hspace{0.17em}300\hspace{0.17em}212\hspace{0.17em}892\hspace{0.17em}225\hspace{0.17em}5}$	$\mathrm{2.020\hspace{0.17em}849\hspace{0.17em}066\hspace{0.17em}807\hspace{0.17em}371\hspace{0.17em}3}$	$\mathrm{1.325\hspace{0.17em}865\hspace{0.17em}758\hspace{0.17em}127\hspace{0.17em}423}$	$\mathrm{5.083\hspace{0.17em}644\hspace{0.17em}512\hspace{0.17em}158\hspace{0.17em}225}$	$\mathrm{1.059\hspace{0.17em}832\hspace{0.17em}938\hspace{0.17em}860\hspace{0.17em}735\hspace{0.17em}7}$		$\mathrm{1.987\hspace{0.17em}296\hspace{0.17em}375\hspace{0.17em}785\hspace{0.17em}886}$	$\mathrm{1.924\hspace{0.17em}395\hspace{0.17em}357\hspace{0.17em}985\hspace{0.17em}260\hspace{0.17em}4}$	$\mathrm{1.810\hspace{0.17em}265\hspace{0.17em}573\hspace{0.17em}809\hspace{0.17em}368\hspace{0.17em}5}$	$5.7$
Dec	$\mathrm{0.743\hspace{0.17em}667\hspace{0.17em}322\hspace{0.17em}538\hspace{0.17em}939\hspace{0.17em}1}$	$\mathrm{0.641\hspace{0.17em}614\hspace{0.17em}311\hspace{0.17em}335\hspace{0.17em}739\hspace{0.17em}5}$	$\mathrm{0.554\hspace{0.17em}421\hspace{0.17em}034\hspace{0.17em}188\hspace{0.17em}418\hspace{0.17em}3}$	$\mathrm{1.672\hspace{0.17em}463\hspace{0.17em}708\hspace{0.17em}556\hspace{0.17em}873\hspace{0.17em}4}$	$\mathrm{0.353\hspace{0.17em}972\hspace{0.17em}893\hspace{0.17em}050\hspace{0.17em}748\hspace{0.17em}6}$		$\mathrm{0.812\hspace{0.17em}426\hspace{0.17em}012\hspace{0.17em}729\hspace{0.17em}449\hspace{0.17em}9}$	$\mathrm{0.863\hspace{0.17em}868\hspace{0.17em}454\hspace{0.17em}458\hspace{0.17em}826\hspace{0.17em}8}$	$\mathrm{0.546\hspace{0.17em}335\hspace{0.17em}440\hspace{0.17em}230\hspace{0.17em}114\hspace{0.17em}4}$	$2.9$
Attn	$\mathrm{1.622\hspace{0.17em}756\hspace{0.17em}290\hspace{0.17em}092\hspace{0.17em}544\hspace{0.17em}5}$	$\mathrm{1.853\hspace{0.17em}022\hspace{0.17em}211\hspace{0.17em}682\hspace{0.17em}523\hspace{0.17em}7}$	$\mathrm{1.750\hspace{0.17em}507\hspace{0.17em}554\hspace{0.17em}412\hspace{0.17em}840\hspace{0.17em}8}$	$\mathrm{3.570\hspace{0.17em}035\hspace{0.17em}656\hspace{0.17em}092\hspace{0.17em}445\hspace{0.17em}6}$	$\mathrm{1.022\hspace{0.17em}157\hspace{0.17em}051\hspace{0.17em}162\hspace{0.17em}584\hspace{0.17em}4}$		$\mathrm{1.823\hspace{0.17em}169\hspace{0.17em}189\hspace{0.17em}348\hspace{0.17em}066\hspace{0.17em}8}$	$\mathrm{2.042\hspace{0.17em}293\hspace{0.17em}802\hspace{0.17em}158\hspace{0.17em}492}$	$\mathrm{1.659\hspace{0.17em}790\hspace{0.17em}860\hspace{0.17em}211\hspace{0.17em}891\hspace{0.17em}2}$	$6.4$
Prepend	$\mathrm{7.547\hspace{0.17em}327\hspace{0.17em}748\hspace{0.17em}583\hspace{0.17em}423}$	$\mathrm{10.988\hspace{0.17em}432\hspace{0.17em}243\hspace{0.17em}838\hspace{0.17em}474}$	$\mathrm{5.836\hspace{0.17em}295\hspace{0.17em}878\hspace{0.17em}299\hspace{0.17em}1}$	$\mathrm{13.160\hspace{0.17em}623\hspace{0.17em}068\hspace{0.17em}414\hspace{0.17em}17}$	$\mathrm{3.908\hspace{0.17em}913\hspace{0.17em}549\hspace{0.17em}412\hspace{0.17em}104}$	$\mathrm{3.436\hspace{0.17em}315\hspace{0.17em}555\hspace{0.17em}400\hspace{0.17em}798\hspace{0.17em}7}$	lt	$\mathrm{9.167\hspace{0.17em}396\hspace{0.17em}158\hspace{0.17em}602\hspace{0.17em}784}$	$\mathrm{5.432\hspace{0.17em}153\hspace{0.17em}976\hspace{0.17em}386\hspace{0.17em}472}$	$17.6$
Emb	$\mathrm{4.605\hspace{0.17em}523\hspace{0.17em}095\hspace{0.17em}176\hspace{0.17em}305}$	$\mathrm{9.291\hspace{0.17em}767\hspace{0.17em}262\hspace{0.17em}879\hspace{0.17em}674}$	$\mathrm{3.671\hspace{0.17em}332\hspace{0.17em}414\hspace{0.17em}719\hspace{0.17em}902\hspace{0.17em}5}$	$\mathrm{16.968\hspace{0.17em}488\hspace{0.17em}425\hspace{0.17em}665\hspace{0.17em}9}$	$\mathrm{2.421\hspace{0.17em}296\hspace{0.17em}905\hspace{0.17em}162\hspace{0.17em}654\hspace{0.17em}5}$	$\mathrm{2.208\hspace{0.17em}329\hspace{0.17em}025\hspace{0.17em}354\hspace{0.17em}667\hspace{0.17em}7}$		$\mathrm{7.272\hspace{0.17em}494\hspace{0.17em}853\hspace{0.17em}071\hspace{0.17em}928}$	$\mathrm{3.801\hspace{0.17em}076\hspace{0.17em}861\hspace{0.17em}876\hspace{0.17em}731}$	$14.0$
Dec	$\mathrm{8.277\hspace{0.17em}131\hspace{0.17em}344\hspace{0.17em}355\hspace{0.17em}158}$	$\mathrm{11.896\hspace{0.17em}042\hspace{0.17em}941\hspace{0.17em}835\hspace{0.17em}896}$	$\mathrm{6.236\hspace{0.17em}987\hspace{0.17em}447\hspace{0.17em}643\hspace{0.17em}494}$	$\mathrm{12.891\hspace{0.17em}424\hspace{0.17em}912\hspace{0.17em}416\hspace{0.17em}221}$	$\mathrm{3.040\hspace{0.17em}638\hspace{0.17em}515\hspace{0.17em}659\hspace{0.17em}347\hspace{0.17em}6}$	$\mathrm{1.559\hspace{0.17em}214\hspace{0.17em}793\hspace{0.17em}471\hspace{0.17em}168\hspace{0.17em}4}$		$\mathrm{10.596\hspace{0.17em}010\hspace{0.17em}897\hspace{0.17em}391\hspace{0.17em}071}$	$\mathrm{6.088\hspace{0.17em}103\hspace{0.17em}747\hspace{0.17em}599\hspace{0.17em}187}$	$18.5$
Attn	$\mathrm{8.322\hspace{0.17em}656\hspace{0.17em}730\hspace{0.17em}311\hspace{0.17em}898}$	$\mathrm{12.012\hspace{0.17em}136\hspace{0.17em}505\hspace{0.17em}207\hspace{0.17em}325}$	$\mathrm{6.547\hspace{0.17em}986\hspace{0.17em}484\hspace{0.17em}687\hspace{0.17em}432}$	$\mathrm{13.156\hspace{0.17em}076\hspace{0.17em}259\hspace{0.17em}113\hspace{0.17em}991}$	$\mathrm{3.923\hspace{0.17em}859\hspace{0.17em}517\hspace{0.17em}510\hspace{0.17em}157}$	$\mathrm{3.164\hspace{0.17em}386\hspace{0.17em}674\hspace{0.17em}912\hspace{0.17em}161}$		$\mathrm{9.857\hspace{0.17em}629\hspace{0.17em}591\hspace{0.17em}881\hspace{0.17em}823}$	$\mathrm{6.238\hspace{0.17em}473\hspace{0.17em}651\hspace{0.17em}431\hspace{0.17em}638}$	$18.7$
Prepend	$\mathrm{8.186\hspace{0.17em}783\hspace{0.17em}472\hspace{0.17em}072\hspace{0.17em}133}$	$\mathrm{12.783\hspace{0.17em}884\hspace{0.17em}671\hspace{0.17em}840\hspace{0.17em}948}$	$\mathrm{5.098\hspace{0.17em}822\hspace{0.17em}863\hspace{0.17em}615\hspace{0.17em}995}$	$\mathrm{13.058\hspace{0.17em}385\hspace{0.17em}816\hspace{0.17em}703\hspace{0.17em}934}$	$\mathrm{3.841\hspace{0.17em}364\hspace{0.17em}780\hspace{0.17em}578\hspace{0.17em}671\hspace{0.17em}5}$	$\mathrm{3.990\hspace{0.17em}072\hspace{0.17em}060\hspace{0.17em}561\hspace{0.17em}763}$	$\mathrm{9.334\hspace{0.17em}562\hspace{0.17em}514\hspace{0.17em}192\hspace{0.17em}513}$	ru	$\mathrm{4.944\hspace{0.17em}812\hspace{0.17em}765\hspace{0.17em}441\hspace{0.17em}785}$	$16.9$
Emb	$\mathrm{6.423\hspace{0.17em}324\hspace{0.17em}843\hspace{0.17em}756\hspace{0.17em}085\hspace{0.17em}5}$	$\mathrm{11.086\hspace{0.17em}007\hspace{0.17em}005\hspace{0.17em}980\hspace{0.17em}98}$	$\mathrm{2.991\hspace{0.17em}629\hspace{0.17em}166\hspace{0.17em}467\hspace{0.17em}560\hspace{0.17em}3}$	$\mathrm{11.942\hspace{0.17em}059\hspace{0.17em}785\hspace{0.17em}363\hspace{0.17em}33}$	$\mathrm{2.891\hspace{0.17em}064\hspace{0.17em}146\hspace{0.17em}685\hspace{0.17em}165\hspace{0.17em}4}$	$\mathrm{2.713\hspace{0.17em}130\hspace{0.17em}539\hspace{0.17em}727\hspace{0.17em}924\hspace{0.17em}6}$	$\mathrm{6.384\hspace{0.17em}892\hspace{0.17em}633\hspace{0.17em}381\hspace{0.17em}313}$		$\mathrm{3.657\hspace{0.17em}126\hspace{0.17em}338\hspace{0.17em}992\hspace{0.17em}887}$	$14.9$
Dec	$\mathrm{9.240\hspace{0.17em}993\hspace{0.17em}787\hspace{0.17em}309\hspace{0.17em}09}$	$\mathrm{13.639\hspace{0.17em}086\hspace{0.17em}171\hspace{0.17em}253\hspace{0.17em}233}$	$\mathrm{5.729\hspace{0.17em}078\hspace{0.17em}463\hspace{0.17em}133\hspace{0.17em}296\hspace{0.17em}5}$	$\mathrm{14.003\hspace{0.17em}735\hspace{0.17em}416\hspace{0.17em}202\hspace{0.17em}8}$	$\mathrm{2.616\hspace{0.17em}308\hspace{0.17em}104\hspace{0.17em}223\hspace{0.17em}372\hspace{0.17em}7}$	$\mathrm{1.552\hspace{0.17em}227\hspace{0.17em}011\hspace{0.17em}926\hspace{0.17em}466\hspace{0.17em}7}$	$\mathrm{9.910\hspace{0.17em}178\hspace{0.17em}858\hspace{0.17em}714\hspace{0.17em}035}$		$\mathrm{5.935\hspace{0.17em}797\hspace{0.17em}388\hspace{0.17em}476\hspace{0.17em}217}$	$19.4$
Attn	$\mathrm{8.745\hspace{0.17em}562\hspace{0.17em}001\hspace{0.17em}518\hspace{0.17em}191}$	$\mathrm{13.752\hspace{0.17em}739\hspace{0.17em}145\hspace{0.17em}507\hspace{0.17em}157}$	$\mathrm{5.586\hspace{0.17em}900\hspace{0.17em}898\hspace{0.17em}372\hspace{0.17em}925}$	$\mathrm{13.760\hspace{0.17em}781\hspace{0.17em}168\hspace{0.17em}520\hspace{0.17em}992}$	$\mathrm{4.133\hspace{0.17em}827\hspace{0.17em}826\hspace{0.17em}188\hspace{0.17em}064}$	$\mathrm{3.636\hspace{0.17em}569\hspace{0.17em}533\hspace{0.17em}175\hspace{0.17em}258}$	$\mathrm{9.639\hspace{0.17em}958\hspace{0.17em}918\hspace{0.17em}862\hspace{0.17em}769}$		$\mathrm{5.920\hspace{0.17em}511\hspace{0.17em}455\hspace{0.17em}681\hspace{0.17em}561}$	$19.2$
Prepend	$\mathrm{5.655\hspace{0.17em}714\hspace{0.17em}658\hspace{0.17em}405\hspace{0.17em}949}$	$\mathrm{8.304\hspace{0.17em}829\hspace{0.17em}759\hspace{0.17em}936\hspace{0.17em}329}$	$\mathrm{4.221\hspace{0.17em}828\hspace{0.17em}293\hspace{0.17em}383\hspace{0.17em}020\hspace{0.17em}5}$	$\mathrm{9.814\hspace{0.17em}406\hspace{0.17em}734\hspace{0.17em}225\hspace{0.17em}692}$	$\mathrm{2.300\hspace{0.17em}556\hspace{0.17em}664\hspace{0.17em}573\hspace{0.17em}756\hspace{0.17em}3}$	$\mathrm{3.128\hspace{0.17em}733\hspace{0.17em}514\hspace{0.17em}864\hspace{0.17em}990\hspace{0.17em}6}$	$\mathrm{6.662\hspace{0.17em}391\hspace{0.17em}738\hspace{0.17em}877\hspace{0.17em}283}$	$\mathrm{6.601\hspace{0.17em}636\hspace{0.17em}812\hspace{0.17em}145\hspace{0.17em}794}$	tr	$13.9$
Emb	$\mathrm{3.163\hspace{0.17em}366\hspace{0.17em}734\hspace{0.17em}167\hspace{0.17em}544}$	$\mathrm{6.474\hspace{0.17em}078\hspace{0.17em}124\hspace{0.17em}680\hspace{0.17em}139\hspace{0.17em}5}$	$\mathrm{3.323\hspace{0.17em}567\hspace{0.17em}666\hspace{0.17em}142\hspace{0.17em}027\hspace{0.17em}5}$	$\mathrm{13.666\hspace{0.17em}011\hspace{0.17em}972\hspace{0.17em}275\hspace{0.17em}493}$	$\mathrm{1.896\hspace{0.17em}340\hspace{0.17em}679\hspace{0.17em}300\hspace{0.17em}77}$	$\mathrm{2.449\hspace{0.17em}724\hspace{0.17em}060\hspace{0.17em}151\hspace{0.17em}728\hspace{0.17em}3}$	$\mathrm{4.665\hspace{0.17em}519\hspace{0.17em}514\hspace{0.17em}179\hspace{0.17em}498}$	$\mathrm{4.314\hspace{0.17em}405\hspace{0.17em}501\hspace{0.17em}303\hspace{0.17em}736}$		$11.1$
Dec	$\mathrm{5.184\hspace{0.17em}810\hspace{0.17em}194\hspace{0.17em}497\hspace{0.17em}06}$	$\mathrm{9.060\hspace{0.17em}894\hspace{0.17em}250\hspace{0.17em}548\hspace{0.17em}909}$	$\mathrm{4.340\hspace{0.17em}068\hspace{0.17em}825\hspace{0.17em}052\hspace{0.17em}572\hspace{0.17em}5}$	$\mathrm{9.687\hspace{0.17em}977\hspace{0.17em}450\hspace{0.17em}768\hspace{0.17em}525}$	$\mathrm{2.554\hspace{0.17em}229\hspace{0.17em}885\hspace{0.17em}708\hspace{0.17em}765\hspace{0.17em}7}$	$\mathrm{1.606\hspace{0.17em}251\hspace{0.17em}980\hspace{0.17em}436\hspace{0.17em}745\hspace{0.17em}7}$	$\mathrm{6.272\hspace{0.17em}970\hspace{0.17em}981\hspace{0.17em}015\hspace{0.17em}550\hspace{0.17em}5}$	$\mathrm{5.919\hspace{0.17em}588\hspace{0.17em}021\hspace{0.17em}825\hspace{0.17em}274}$		$15.5$
Attn	$\mathrm{5.198\hspace{0.17em}330\hspace{0.17em}448\hspace{0.17em}941\hspace{0.17em}991\hspace{0.17em}5}$	$\mathrm{8.856\hspace{0.17em}113\hspace{0.17em}699\hspace{0.17em}978\hspace{0.17em}494}$	$\mathrm{4.676\hspace{0.17em}246\hspace{0.17em}100\hspace{0.17em}359\hspace{0.17em}614}$	$\mathrm{10.365\hspace{0.17em}935\hspace{0.17em}784\hspace{0.17em}865\hspace{0.17em}913}$	$\mathrm{3.397\hspace{0.17em}218\hspace{0.17em}151\hspace{0.17em}169\hspace{0.17em}341}$	$\mathrm{3.202\hspace{0.17em}720\hspace{0.17em}158\hspace{0.17em}623\hspace{0.17em}593}$	$\mathrm{6.041\hspace{0.17em}666\hspace{0.17em}490\hspace{0.17em}731\hspace{0.17em}7}$	$\mathrm{6.018\hspace{0.17em}216\hspace{0.17em}234\hspace{0.17em}009\hspace{0.17em}529}$		$16.2$
Prepend	$\mathrm{4.549\hspace{0.17em}090\hspace{0.17em}162\hspace{0.17em}049\hspace{0.17em}668}$	$\mathrm{7.486\hspace{0.17em}307\hspace{0.17em}931\hspace{0.17em}729\hspace{0.17em}135}$	$\mathrm{3.770\hspace{0.17em}714\hspace{0.17em}446\hspace{0.17em}238\hspace{0.17em}774\hspace{0.17em}7}$	$\mathrm{9.479\hspace{0.17em}953\hspace{0.17em}037\hspace{0.17em}856\hspace{0.17em}694}$	$\mathrm{2.638\hspace{0.17em}014\hspace{0.17em}194\hspace{0.17em}315\hspace{0.17em}829}$	$\mathrm{2.672\hspace{0.17em}526\hspace{0.17em}090\hspace{0.17em}032\hspace{0.17em}915\hspace{0.17em}3}$	$\mathrm{6.211\hspace{0.17em}728\hspace{0.17em}742\hspace{0.17em}744\hspace{0.17em}226}$	$\mathrm{5.955\hspace{0.17em}644\hspace{0.17em}043\hspace{0.17em}356\hspace{0.17em}087}$	$\mathrm{3.521\hspace{0.17em}520\hspace{0.17em}799\hspace{0.17em}699\hspace{0.17em}985}$	zh
Emb	$\mathrm{3.666\hspace{0.17em}192\hspace{0.17em}298\hspace{0.17em}862\hspace{0.17em}674\hspace{0.17em}7}$	$\mathrm{6.859\hspace{0.17em}104\hspace{0.17em}781\hspace{0.17em}838\hspace{0.17em}210\hspace{0.17em}5}$	$\mathrm{2.674\hspace{0.17em}429\hspace{0.17em}552\hspace{0.17em}635\hspace{0.17em}564\hspace{0.17em}4}$	$\mathrm{10.666\hspace{0.17em}566\hspace{0.17em}622\hspace{0.17em}674\hspace{0.17em}795}$	$\mathrm{2.515\hspace{0.17em}771\hspace{0.17em}928\hspace{0.17em}963\hspace{0.17em}229}$	$\mathrm{2.005\hspace{0.17em}259\hspace{0.17em}946\hspace{0.17em}865\hspace{0.17em}438\hspace{0.17em}6}$	$\mathrm{5.263\hspace{0.17em}448\hspace{0.17em}807\hspace{0.17em}865\hspace{0.17em}764}$	$\mathrm{5.747\hspace{0.17em}389\hspace{0.17em}057\hspace{0.17em}261\hspace{0.17em}707\hspace{0.17em}5}$	$\mathrm{3.204\hspace{0.17em}311\hspace{0.17em}143\hspace{0.17em}573\hspace{0.17em}751\hspace{0.17em}7}$
Dec	$\mathrm{5.265\hspace{0.17em}268\hspace{0.17em}172\hspace{0.17em}943\hspace{0.17em}661}$	$\mathrm{7.902\hspace{0.17em}216\hspace{0.17em}530\hspace{0.17em}771\hspace{0.17em}674}$	$\mathrm{4.032\hspace{0.17em}513\hspace{0.17em}207\hspace{0.17em}378\hspace{0.17em}797\hspace{0.17em}5}$	$\mathrm{10.460\hspace{0.17em}187\hspace{0.17em}521\hspace{0.17em}640\hspace{0.17em}332}$	$\mathrm{2.098\hspace{0.17em}355\hspace{0.17em}496\hspace{0.17em}752\hspace{0.17em}651\hspace{0.17em}6}$	$\mathrm{1.196\hspace{0.17em}057\hspace{0.17em}955\hspace{0.17em}916\hspace{0.17em}489\hspace{0.17em}2}$	$\mathrm{6.165\hspace{0.17em}459\hspace{0.17em}824\hspace{0.17em}001\hspace{0.17em}96}$	$\mathrm{7.134\hspace{0.17em}309\hspace{0.17em}519\hspace{0.17em}088\hspace{0.17em}071}$	$\mathrm{4.860\hspace{0.17em}487\hspace{0.17em}829\hspace{0.17em}901\hspace{0.17em}768}$
Attn	$\mathrm{5.212\hspace{0.17em}108\hspace{0.17em}723\hspace{0.17em}494\hspace{0.17em}992\hspace{0.17em}5}$	$\mathrm{7.661\hspace{0.17em}761\hspace{0.17em}530\hspace{0.17em}100\hspace{0.17em}979}$	$\mathrm{4.177\hspace{0.17em}677\hspace{0.17em}962\hspace{0.17em}192\hspace{0.17em}277}$	$\mathrm{11.000\hspace{0.17em}200\hspace{0.17em}287\hspace{0.17em}921\hspace{0.17em}622}$	$\mathrm{3.676\hspace{0.17em}491\hspace{0.17em}942\hspace{0.17em}279\hspace{0.17em}176}$	$\mathrm{2.401\hspace{0.17em}345\hspace{0.17em}432\hspace{0.17em}800\hspace{0.17em}162}$	$\mathrm{6.579\hspace{0.17em}181\hspace{0.17em}275\hspace{0.17em}629\hspace{0.17em}187}$	$\mathrm{6.420\hspace{0.17em}570\hspace{0.17em}903\hspace{0.17em}718\hspace{0.17em}975}$	$\mathrm{4.437\hspace{0.17em}867\hspace{0.17em}154\hspace{0.17em}174\hspace{0.17em}928}$

Full text

1 Introduction

Multi-lingual translation models, which can map from multiple source languages into multiple target languages, have recently received significant attention because of the potential for positive transfer between high- and low-resource language pairs, and because of possible efficiency gains enabled by translation models which share parameters across many languages Dong et al. (2015); Ha et al. (2016); Firat et al. (2016); Johnson et al. (2016); Blackwood et al. (2018); Sachan and Neubig (2018); Aharoni et al. (2019). Multi-lingual models that share parameters across languages can also perform zero-shot translation, translating between language pairs for which no parallel training data is available Wu et al. (2016); Ha et al. (2016); Johnson et al. (2016).

Although multi-task models have recently been shown to achieve positive transfer for some combinations of NLP tasks, in the context of MT, multi-lingual models do not universally outperform models trained to translate in a single direction when sufficient training data is available. However, the ability to do zero-shot translation may be of practical importance in many cases, as parallel training data is not available for most language pairs Wu et al. (2016); Johnson et al. (2016); Aharoni et al. (2019). Therefore, small decreases in the performance of supervised pairs may be admissible if the corresponding gain in zero-shot performance is large. In addition, zero-shot translation can be used to generate synthetic training data for low- or zero- resource language pairs, making it a practical alternative to the bootstrapping by back-translation approach that has recently been used to build completely unsupervised MT systems Firat et al. (2016); Artetxe et al. (2018); Lample et al. (2018a, b). Therefore, understanding the trade-offs between different methods of constructing multi-lingual MT systems is still an important line of research.

Deep sequence-to-sequence models have become the established state-of-the-art for machine translation. The dominant paradigm continues to be models divided into roughly three high-level components: embeddings, which map discrete tokens into real-valued vectors, encoders, which map sequences of vectors into an intermediate representation, and decoders, which use the representation from an encoder, combined with a dynamic representation of the current state, and output a sequence of tokens in the target language conditioned upon the encoder’s representation of the input. For multi-lingual systems, any combination of embedding, encoder and/or decoder parameters can potentially be shared by groups of tasks, or duplicated and kept private for each task.

Our work builds upon recent research on many-to-one, one-to-many, and many-to-many translation models. We are interested in evaluating many-to-many models under realistic conditions, including:

A highly imbalanced amount of training data available for different language pairs. 2. 2.

A very diverse set of source and target languages. 3. 3.

Training and evaluation data from many domains.

We focus on multi-layer transformer models Vaswani et al. (2017), which achieve state-of-the-art performance on large-scale MT and NLP tasks Devlin et al. (2018); Bojar et al. (2018). The decoder component of the transformer is visualized in figure 1. We study four ways of building multi-lingual translation models. Importantly, all of the models we study can do zero-shot translation: translating between language pairs for which no parallel data was seen at training time. The models use training data from 11 distinct languages111cs, de, en, fi, fr, gu, kk, lt, ru, tr and zh, with supervised data available from the WMT19 news-translation task for 22 of the 110 unique translation directions222Note we do not consider auto-encoding, thus the number of translation directions is $11^{2}-11=110$.. This leaves 88 translation directions for which no parallel data is available. We try to evaluate zero-shot translation performance on all of these additional directions, using both gold parallel data, and evaluations based on pivoting or multi-hop translation.

Target Language Specification

Although the embedding and encoder parameters of a multi-lingual system may be shared across all languages without any special modification to the model, decoding from a multi-lingual model requires a means of specifying the desired output language. Previous work has accomplished this in different ways, including:

• •

pre-pending a special target-language token to the input Wu et al. (2016)

• •

using an additional embedding vector for the target language Lample and Conneau (2019)

• •

using unique decoders for each target language Luong et al. (2016); Firat et al. (2016)

• •

partially sharing some of the decoder parameters while keeping others unique to each target language Sachan and Neubig (2018); Blackwood et al. (2018).

However, to the best of our knowledge, no side-by-side comparison of these approaches has been conducted. We therefore train models which are identical except for the way that decoding into different target languages is handled, and conduct a large-scale evaluation. We use only the language pairs and official parallel data released by the WMT task organisers, meaning that all of our systems correspond to the constrained setting of the WMT shared task, and our experimental settings should thus be straightforward to replicate.

2 Multi-Task Translation Models

This section discusses the key components of the transformer-based NMT model, focusing on the various ways to enable translation into many target languages. We use the terms source/target task and language interchangeably, to emphasize our view that multi-lingual NMT is one instantiation of the more general case of multi-task sequence to sequence learning.

2.1 Shared Encoders and Embeddings

In this work, we are only interested in ways of providing target task information to the model – information about the source task is never given explicitly, and encoder parameters are always fully shared across all tasks. The segmentation model and embedding parameters are also shared between all source and target tasks (see below for more details).

2.2 Multi-lingual Decoder Configurations

Figure 1 visualizes the decoder component of the transformer model, with dashed and dotted lines indicating the parameter sets that we can replicate or share across target tasks.

2.2.1 Target Task Tokens (Prepend)

Wu et al. (2016) showed that, as long as a mechanism exists for specifying the target task, it is possible to share the decoder module’s parameters across all tasks. In the case where all parameters are shared, the decoder model must learn to operate in a number of distinct modes which are triggered by some variation in the input. A simple way to achive this variation is by pre-pending a special "task-token" to each input. We refer to this method as Prepend.

2.2.2 Task Embeddings (Emb)

An alternative to the use of a special task token is to treat the target task as an additional input feature, and to train a unique embedding for each target task Lample and Conneau (2019), which is combined with the source input. This technique has the advantage of explicitly decoupling target task information from source task input, introducing a relatively small number of additional parameters. This approach can also be seen as adding an additional token-level feature which is the same for all tokens in a sequence Sennrich and Haddow (2016). We refer to this setting as Emb.

2.2.3 Task-specific Decoders (Dec)

In general, any subset of decoder parameters may be replicated for each target language, resulting in parameter sets which are specific to each target task. At one extreme, the entire decoder module may be replicated for each target language, a setting which we label Dec Dong et al. (2015).

2.2.4 Task-specific Attention (Attn)

An approach somewhere in-between Emb and Dec is to partially share some of the decoder parameters, while keeping others unique to each task. Recent work proposed creating unique attention modules for every target task, while sharing the other decoder parameters Sachan and Neubig (2018); Blackwood et al. (2018). The implementation of their approaches differ significantly – we propose to create completely unique attention parameters for each task. This means that for each of our 11 languages, we have unique context- and self-attention parameters in each layer of the transformer decoder. We refer to this setting as Attn.

3 Experiments

All experiments are conducted using the transformer-base configuration of Vaswani et al. (2017) with the relevant modifications for each system discussed in the previous section. We use a shared sentencepiece333https://github.com/google/sentencepiece segmentation model with 32000 pieces. We use all available parallel data from the WMT19 news-translation task for training, with the exception of commoncrawl, which we found to be very noisy after manually checking a sample of the data, and paracrawl, which we use only for en-fi and en-lt444Turkish (tr) is included from the 2018 language pairs because the task-organizers suggest the possibility of using tr data to improve kk performance.

We train each model on two P100 GPUs with an individual batch size of up to 2048 tokens. Gradients are accumulated over 8 mini-batches and parameters are updated synchronously, meaning that our effective batch size is $2*2048*4=16384$ tokens per iteration. Because the task pair for each mini-batch is sampled according to our policy weights and (fixed) random seed, and each iteration consists of 8 unique mini-batches, a single parameter update can potentially contain information from up to 8 unique task pairs. We train each model for 100,000 iterations without early stopping, which takes about 40 hours per model. When evaluating we always use the final model checkpoint (i.e. the model parameters saved after 100,000 iterations). We use our in-house research NMT system, which is heavily based upon OpenNMT-py Klein et al. (2017).

The sampling policy weights were specified manually by looking at the amount of available data for each pair, and estimating the difficulty of each translation direction. The result of the sampling policy is that lower resource language pairs are upsampled significantly. Table 1 summarizes the statistics for each language pair. Note that the data in each row represents a pair of tasks, i.e. the total number of segments seen for en-cs is split evenly between en$\rightarrow$cs, and cs$\rightarrow$en. Because we train for only 100,000 iterations, we do not see all of the available training data for some high-resource language pairs.

With the exception of the Prepend system, the input to each model is identical. Each experimental setting is mutually exclusive, i.e. in the Emb setting we do not prepend task tokens, and in the Attn setting we do not use task embeddings.

Figure 2 plots the validation performance during training on one of our validation datasets. The language embeddings from the Emb system are visualized in figure 3.

3.1 Results

We conduct four different evaluations of the performance of our models. First, we check performance on the 22 supervised pairs using dev and test sets from the WMT shared task. We then try to evaluate zero-shot translation performance in several ways. We use the TED talks multi-parallel dataset Ye et al. (2018) to create gold sets for all zero-shot pairs that occur in the TED talks corpus, and evaluate on those pairs. We also try two ways of evaluating zero-shot translation without gold data. In the first, we do round-trip translation $\textsc{SRC}\rightarrow\textsc{Pivot}\rightarrow\savestack{\tmpbox}{\stretchto{\scaleto{\scalerel*[\widthof{\textsc{SRC}}]{\kern 0.1pt\mathchar 866\relax\kern 0.1pt}{\rule{0.0pt}{702.78297pt}}}{}}{2.4ex}}\stackon[-6.9pt]{\textsc{SRC}}{\tmpbox}$, and measure performance on the $(\savestack{\tmpbox}{\stretchto{\scaleto{\scalerel*[\widthof{\textsc{SRC}}]{\kern 0.1pt\mathchar 866\relax\kern 0.1pt}{\rule{0.0pt}{702.78297pt}}}{}}{2.4ex}}\stackon[-6.9pt]{\textsc{SRC}}{\tmpbox},\textsc{SRC})$ pair – this method is labeled Zero-Shot Pivot. In the second, we use parallel evaluation datasets from the WMT shared tasks (consisting of $(\textsc{SRC},\textsc{REF})$ pairs), and translate $\textsc{SRC}\rightarrow\textsc{Pivot}\rightarrow\savestack{\tmpbox}{\stretchto{\scaleto{\scalerel*[\widthof{\textsc{TRG}}]{\kern 0.1pt\mathchar 866\relax\kern 0.1pt}{\rule{0.0pt}{702.78297pt}}}{}}{2.4ex}}\stackon[-6.9pt]{\textsc{TRG}}{\tmpbox}$, then measure performance on the resulting $(\savestack{\tmpbox}{\stretchto{\scaleto{\scalerel*[\widthof{\textsc{TRG}}]{\kern 0.1pt\mathchar 866\relax\kern 0.1pt}{\rule{0.0pt}{702.78297pt}}}{}}{2.4ex}}\stackon[-6.9pt]{\textsc{TRG}}{\tmpbox},\textsc{REF})$ pairs (see below for more details), where the pivot and target language pair is a zero-shot translation task – this method is labeled Zero-Shot Parallel Pivot555For the Zero-Shot Pivot and Zero-shot parallel pivot evaluations we use the first 1000 segments of each dataset, because we need to translate twice for every possible pair.

Table 2 lists the WMT evaluation dataset that we use for each language pair. In the Zero-Shot Pivot setting, the reference side of the dataset is used as input.

Table 3 shows global results for all parallel tasks and all zero-shot tasks, by system. Global scores are obtained by concatenating the segmented outputs for each translation direction, and computing the BLEU score against the corresponding concatenated, segmented reference translations. The results in table 3 are thus tokenized BLEU scores.

3.2 Parallel Tasks

Table 4 lists results for all supervised task pairs from WMT 2019. For each pair, we report BLEU scores on de-tokenized output, and compute scores using sacrebleu 666BLEU+case.mixed+ lang.<src-lang>-<trg-lang>+ numrefs.1+smooth.exp+tok.<trg-lang>+ version.1.2.19. Therefore, we expect BLEU scores to be equivalent to those used in the WMT automatic evaluation.

We note that across all but the lowest-resource tasks, the model with a unique decoder for each language outperforms all others. However, for en$\rightarrow$gu and en$\rightarrow$kk, the lowest-resource translation directions, the unique decoder model fails completely, probably because the unique parameters for kk and gu were not updated by a sufficient number of mini-batches (approximately 15,600 for en$\rightarrow$gu and 14,800 for en$\rightarrow$kk).

3.3 Zero-shot Translation Tasks

In order to test our models in the zero-shot setting, we first create a multi-parallel dataset from the from the TED Talks multi-parallel corpus Ye et al. (2018), which has recently been used for the training and evaluation of multi-lingual models. We filter the dev and test sets of this corpus to find segments which have translations for all of en, fr, ru, tr, de, cs, lt, fi, and are at least 20 characters long, resulting in 606 segments. Because this corpus is pre-processed, we first de-tokenize and de-escape punctuation using sacremoses777https://github.com/alvations/sacremoses. We then evaluate zero-shot translation for all possible pairs which do not occur in our parallel training data, aggregate results are shown in the second row of table 3.

We then adapt an evaluation technique that has recently been used for unsupervised MT – we translate from the source language into a pivot language, then back into the source language, and evaluate the score of the resulting source-language hypotheses against the original source Lample et al. (2018a). This technique allows us to evaluate for all possible translation directions in our multi-directional model.

Aware of the risk that the model simply copies through the original source segment instead of translating, we assert that at least 95% of pivot translations’ language code is correctly detected by langid888https://github.com/saffsd/langid.py, and pairs which do not meet this criteria for any system are removed from the evaluation for all systems (not just for the system that failed). For all models except Emb only ru$\rightarrow$kk$\rightarrow$ru fi$\rightarrow$lt$\rightarrow$fi, and zh$\rightarrow$gu$\rightarrow$zh failed this test, but for the Emb model 31 of the 110 translation directions failed (see tables 6 and 7(in appendix)999We conduct round trip translation on all 110 directions, but we only use directions that are (1) not available in the parallel training data, and (2) pass the language identification test to compute the global zero-shot translation performance.. This result indicates that models which use language embeddings may have a more "fuzzy" representation of the output task, and are much more prone to copying than other approaches to multi-lingual MT.

Finally, we conduct the Zero-Shot Parallel Pivot evaluation using the same datasets in table 2 by translating from EN (or DE in the case of *-FR) to each possible pivot language, and then from the pivot language into the target language. Compared to the Zero-Shot Pivot setting, this evaluation should help to protect against the risk of copying, because source and reference segments are not from the same language. Aggregate results for this setting are shown in the third row of table 3, full results in table 8 in appendix.

3.4 Discussion

Our results show that a models with either (1) a completely unique decoders for each target language or (2) unique decoder attention parameters for each target language clearly outperform models with fully shared decoder parameters.

As shown in table 3, the Zero-Shot Pivot evaluation is the outlier in our results, with the Emb system outperforming the others. Even for the languages which passed the language identification filter used in this evaluation, we suspect that some copying is occurring for the Emb system, because of the mismatch in results between the Zero-Shot Pivot task and the Supervised, Zero-Shot TED, and Zero-shot Parallel Pivot tasks (see table 3). Since the ranking of the models according to the Zero-Shot Parallel Pivot evaluation is well aligned with the Zero-Shot TED and Supervised evaluations which use gold parallel evaluation data, we believe that this method is effective for zero-shot evaluation of translation quality for language pairs where no gold data is available.

It is plausible that the language-independence of encoder output could be correlated with the amount of sharing in the decoder module. Because most non-English target tasks only have parallel training data in English, a unique decoder for those tasks only needs to learn to decode from English, not from every possible source task. However, our results show that the Attn model, which partially shares parameters across target languages only slightly outperforms the Dec model globally, because of the improved performance of the Attn model on the lowest-resource tasks (Table 4, Table 7 (in appendix)). Thus we conclude that multi-lingual encoders still learn to share information across languages, even when trained using decoders that are unique to each target task.

4 Related Work

Dong et al. (2015); Firat et al. (2016); Ha et al. (2016); Johnson et al. (2016) and others have shown that multi-way NMT systems can be created with minimal modification to the approach used for single-language-pair systems. Johnson et al. (2016) showed that simply prepending a target-task token to source inputs is enough to enable zero-shot translation between language pairs for which no parallel training data is available.

Our work is most similar to Sachan and Neubig (2018), where several different strategies for sharing decoder parameters are investigated for one-to-many translation models. However, their evaluation setting is constrained to one-to-many models which translate from English into two target languages, whereas our setting is more ambitious, performing multi-way translation between 11 languages. Blackwood et al. (2018) showed that using separate attention parameters for each task can improve the performance of multi-task MT models – this work was the inspiration for the Attn setting in our experiments.

Several recent papers focus specifically upon improving the zero-shot performance of multi-lingual MT models Chen et al. (2017); Arivazhagan et al. (2019); Gu et al. (2019); Lu et al. (2018); Al-Shedivat and Parikh (2019); Sestorain et al. (2019).

Concurrently with this work, Aharoni et al. (2019) evaluated a multiway MT system on a large number of language pairs using the TED talks corpus. However, they focus upon EN-* and *-EN, and do not test different model variants.

5 Conclusions and Future Work

We have presented results which are consistent with recent smaller-scale evaluations of multi-lingual MT systems, showing that assigning unique attention parameters to each target language in a multi-lingual NMT system is optimal when evaluating such a system globally. However, when evaluated on the individual task level, models which have unique decoder parameters for every target task tend to outperform other configurations, except when the amount of available training data is extremely small. We have also introduced two methods of evaluating zero-shot translation performance when parallel data is not available, and we conducted a large-scale evaluation of translation performance across all possible translation directions in the constrained setting of the WMT19 news-translation task.

In future work, we hope to continue studying how multi-lingual translation systems scale to realistic volumes of training data and large numbers of source and target tasks.