Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence

TL;DR

This paper explores converting pretrained non-recurrent language models into depth-recurrent models using a curriculum of recurrences, leading to improved performance and reduced computational costs.

Contribution

It introduces a method to retrofit existing pretrained models into depth-recurrent models with a curriculum approach, enhancing efficiency and performance.

Findings

Recurrent models outperform non-recurrent ones at the same compute budget.

Curriculum-based training preserves performance while reducing total computational cost.

Recurrent models show improved results in mathematical tasks.

Abstract

Recent advances in depth-recurrent language models show that recurrence can decouple train-time compute and parameter count from test-time compute. In this work, we study how to convert existing pretrained non-recurrent language models into depth-recurrent models. We find that using a curriculum of recurrences to increase the effective depth of the model over the course of training preserves performance while reducing total computational cost. In our experiments, on mathematics, we observe that converting pretrained models to recurrent ones results in better performance at a given compute budget than simply post-training the original non-recurrent language model.