Extrapolating Multilingual Understanding Models as Multilingual Generators

TL;DR

This paper introduces a novel method called Semantic-Guided Alignment-then-Denoising (SGA) to adapt multilingual understanding models into effective multilingual generators, achieving significant improvements in various generation tasks.

Contribution

It proposes the SGA approach to transform encoder-based models into generators with minimal additional parameters, outperforming traditional initialization methods.

Findings

SGA outperforms initialization-based methods in translation and question/story generation.

XLM-R shows strong zero-shot performance but lags behind mBART in supervised tasks.

More research is needed to enhance understanding models for generation capabilities.

Abstract

Multilingual understanding models (or encoder-based), pre-trained via masked language modeling, have achieved promising results on many language understanding tasks (e.g., mBERT). However, these non-autoregressive (NAR) models still struggle to generate high-quality texts compared with autoregressive (AR) models. Considering that encoder-based models have the advantage of efficient generation and self-correction abilities, this paper explores methods to empower multilingual understanding models the generation abilities to get a unified model. Specifically, we start from a multilingual encoder (XLM-R) and propose a \textbf{S}emantic-\textbf{G}uided \textbf{A}lignment-then-Denoising (SGA) approach to adapt an encoder to a multilingual generator with a small number of new parameters. Experiments show that the proposed approach is an effective adaption method, outperforming widely-used initialization-based methods with gains of 9.4 BLEU on machine translation, 8.1 Rouge-L on question generation, and 5.5 METEOR on story generation on XLM-R$_{large}$. On the other hand, we observe that XLM-R is still inferior to mBART in supervised settings despite better results on zero-shot settings, indicating that more exploration is required to make understanding models strong generators.