Dialogue Is Not Enough to Make a Communicative BabyLM (But Neither Is Developmentally Inspired Reinforcement Learning)

TL;DR

This paper explores the limitations of pre-training small language models solely on dialogue data, showing that while they excel in dialogue tasks, they underperform on standard benchmarks, with fine-tuning strategies offering mixed results.

Contribution

It introduces a dialogue-focused pre-training approach and evaluates various fine-tuning methods, highlighting challenges in achieving general language understanding from dialogue data alone.

Findings

Models excel at dialogue continuation prediction.

Pre-training on dialogue data alone underperforms on standard benchmarks.

DPO fine-tuning improves dialogue benchmark performance.

Abstract

We investigate whether pre-training exclusively on dialogue data results in formally and functionally apt small language models. Based on this pre-trained llamalogue model, we employ a variety of fine-tuning strategies to enforce "more communicative" text generations by our models. Although our models underperform on most standard BabyLM benchmarks, they excel at dialogue continuation prediction in a minimal pair setting. While PPO fine-tuning has mixed to adversarial effects on our models, DPO fine-tuning further improves their performance on our custom dialogue benchmark.