BabyLM Challenge: Exploring the Effect of Variation Sets on Language Model Training Efficiency

TL;DR

This study investigates how Variation Sets, common in child-directed speech, affect the training efficiency of language models like GPT-2, revealing that their impact varies across benchmarks and training conditions.

Contribution

It introduces the use of artificial Variation Sets in training data and analyzes their effects on language model performance within the BabyLM Challenge.

Findings

Variation Sets improve BLiMP and GLUE scores.

Impact of Variation Sets depends on training epochs and data presentation order.

Results indicate potential benefits of Variation Sets for language model training.

Abstract

While current large language models have achieved a remarkable success, their data efficiency remains a challenge to overcome. Recently it has been suggested that child-directed speech (CDS) can improve training data efficiency of modern language models based on Transformer neural networks. However, it is not yet understood which specific properties of CDS are effective for training these models. In the context of the BabyLM Challenge, we focus on Variation Sets (VSs), sets of consecutive utterances expressing a similar intent with slightly different words and structures, which are ubiquitous in CDS. To assess the impact of VSs on training data efficiency, we augment CDS data with different proportions of artificial VSs and use these datasets to train an auto-regressive model, GPT-2. We find that the best proportion of VSs depends on the evaluation benchmark: BLiMP and GLUE scores benefit from the presence of VSs, but EWOK scores do not. Additionally, the results vary depending on multiple factors such as the number of epochs and the order of utterance presentation. Taken together, these findings suggest that VSs can have a beneficial influence on language models, while leaving room for further investigation.