CLASS-IT: Conversational and Lecture-Aligned Small-Scale Instruction Tuning for BabyLMs

TL;DR

This study explores instruction tuning for small-scale language models, comparing conversational and question-answering datasets in different curricula, revealing modest fine-tuning gains but limited zero-shot transfer, highlighting challenges in low-resource LM adaptation.

Contribution

It introduces a systematic comparison of instruction tuning strategies for small LMs, emphasizing curriculum design and its impact on fine-tuning and zero-shot performance.

Findings

Sequential curricula outperform merged data in fine-tuning.

Instruction tuning provides small but consistent gains in fine-tuning.

Limited transfer of improvements to zero-shot tasks.

Abstract

This work investigates whether small-scale LMs can benefit from instruction tuning. We compare conversational and question-answering instruction tuning datasets, applied either in a merged or sequential curriculum, using decoder-only models with 100M and 140M parameters. Evaluation spans both fine-tuning (SuperGLUE) and zero-shot (BLiMP, EWoK, WUGs, entity tracking, and psycholinguistic correlation) settings. Results show that instruction tuning yields small but consistent gains in fine-tuning scenarios, with sequential curricula outperforming merged data; however, improvements do not consistently transfer to zero-shot tasks, suggesting a trade-off between interaction-focused adaptation and broad linguistic generalization. These results highlight both the potential and the constraints of adapting human-inspired learning strategies to low-resource LMs, and point toward hybrid, curriculum-based approaches for enhancing generalization under ecological training limits.