ToddlerBERTa: Exploiting BabyBERTa for Grammar Learning and Language Understanding

TL;DR

ToddlerBERTa, a smaller language model inspired by BabyBERTa, demonstrates strong language understanding capabilities across various benchmarks, highlighting the effectiveness of hyperparameter tuning and limited data training.

Contribution

Introduces ToddlerBERTa, a new language model that performs competitively with larger models despite limited training data and explores hyperparameter impacts.

Findings

Smaller models can excel in specific tasks.

Larger models perform well with more data.

ToddlerBERTa rivals state-of-the-art models.

Abstract

We present ToddlerBERTa, a BabyBERTa-like language model, exploring its capabilities through five different models with varied hyperparameters. Evaluating on BLiMP, SuperGLUE, MSGS, and a Supplement benchmark from the BabyLM challenge, we find that smaller models can excel in specific tasks, while larger models perform well with substantial data. Despite training on a smaller dataset, ToddlerBERTa demonstrates commendable performance, rivalling the state-of-the-art RoBERTa-base. The model showcases robust language understanding, even with single-sentence pretraining, and competes with baselines that leverage broader contextual information. Our work provides insights into hyperparameter choices, and data utilization, contributing to the advancement of language models.