Transfer of Structural Knowledge from Synthetic Languages

TL;DR

This paper investigates how training on synthetic languages can improve transfer learning to English, analyzing embedding structures, introducing a new synthetic language, and proposing Tiny-Cloze Benchmark for better evaluation.

Contribution

It introduces a new synthetic language that enhances transfer to English and presents Tiny-Cloze Benchmark for evaluating models on linguistic tasks.

Findings

Fine-tuning on synthetic languages improves English transfer performance.

The new synthetic language outperforms previous ones in transfer tasks.

Tiny-Cloze Benchmark provides more informative evaluation for less powerful models.

Abstract

This work explores transfer learning from several synthetic languages to English. We investigate the structure of the embeddings in the fine-tuned models, the information they contain, and the capabilities of the fine-tuned models on simple linguistic tasks. We also introduce a new synthetic language that leads to better transfer to English than the languages used in previous research. Finally, we introduce Tiny-Cloze Benchmark - a new synthetic benchmark for natural language understanding that is more informative for less powerful models. We use Tiny-Cloze Benchmark to evaluate fine-tuned models in several domains demonstrating that fine-tuning on a new synthetic language allows for better performance on a variety of tasks.