A surprisal oracle for when every layer counts

TL;DR

This paper introduces an improved active curriculum learning method for language models, dynamically guiding training based on model uncertainty, leading to better performance on common-sense and world-knowledge tasks.

Contribution

It presents an updated, more dynamic uncertainty model for ACLM and applies it to the ELC-BERT model for the BabyLM 2024 task.

Findings

Outperforms BabyLM 2024 baselines on common-sense tasks

Underperforms on fine-grained grammatical inference

Improves curriculum construction through a dynamic similarity model

Abstract

Active Curriculum Language Modeling (ACLM; Hong et al., 2023) is a learner directed approach to training a language model. We proposed the original version of this process in our submission to the BabyLM 2023 task, and now we propose an updated ACLM process for the BabyLM 2024 task. ACLM involves an iteratively- and dynamically-constructed curriculum informed over the training process by a model of uncertainty; other training items that are similarly uncertain to a least certain candidate item are prioritized. Our new process improves the similarity model so that it is more dynamic, and we run ACLM over the most successful model from the BabyLM 2023 task: ELC-BERT (Charpentier and Samuel, 2023). We find that while our models underperform on fine-grained grammatical inferences, they outperform the BabyLM 2024 official base-lines on common-sense and world-knowledge tasks. We make our code available at https: //github.com/asayeed/ActiveBaby.