As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languages

TL;DR

This paper presents a method to adapt English GPT-2 models to Italian and Dutch by retraining lexical embeddings, enabling realistic sentence generation in these languages with minimal additional training.

Contribution

It introduces a novel lexical embedding adaptation technique that allows repurposing English GPT-2 models for other languages without full retraining.

Findings

Generated sentences in Italian and Dutch are realistic and comparable to fully trained models.

The method minimizes training time and preserves original model knowledge.

Relearned lexical embeddings align with original English embeddings.

Abstract

Large generative language models have been very successful for English, but other languages lag behind, in part due to data and computational limitations. We propose a method that may overcome these problems by adapting existing pre-trained models to new languages. Specifically, we describe the adaptation of English GPT-2 to Italian and Dutch by retraining lexical embeddings without tuning the Transformer layers. As a result, we obtain lexical embeddings for Italian and Dutch that are aligned with the original English lexical embeddings. Additionally, we scale up complexity by transforming relearned lexical embeddings of GPT-2 small to the GPT-2 medium embedding space. This method minimises the amount of training and prevents losing information during adaptation that was learned by GPT-2. English GPT-2 models with relearned lexical embeddings can generate realistic sentences in Italian and Dutch. Though on average these sentences are still identifiable as artificial by humans, they are assessed on par with sentences generated by a GPT-2 model fully trained from scratch.