Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models

TL;DR

This study explores how pretraining on artificial languages with structural features similar to natural language can transfer knowledge to natural language processing tasks, revealing insights into neural encoders' understanding of language structure.

Contribution

The paper introduces a method of pretraining with artificial languages to analyze transferability of structural knowledge in neural encoders, providing new insights into language understanding.

Findings

Pretraining on artificial languages with nesting structures transfers some knowledge to natural language tasks.

Transferability is linked to the amount of contextual information encoded.

Knowledge transferred includes position-aware context dependence.

Abstract

We investigate what kind of structural knowledge learned in neural network encoders is transferable to processing natural language. We design artificial languages with structural properties that mimic natural language, pretrain encoders on the data, and see how much performance the encoder exhibits on downstream tasks in natural language. Our experimental results show that pretraining with an artificial language with a nesting dependency structure provides some knowledge transferable to natural language. A follow-up probing analysis indicates that its success in the transfer is related to the amount of encoded contextual information and what is transferred is the knowledge of position-aware context dependence of language. Our results provide insights into how neural network encoders process human languages and the source of cross-lingual transferability of recent multilingual language models.