Self-training Language Models for Arithmetic Reasoning

TL;DR

This paper investigates how self-training with automated feedback can enhance arithmetic reasoning in language models without additional annotated data, showing significant improvements in accuracy across multiple datasets.

Contribution

It demonstrates that self-training with automated feedback improves arithmetic reasoning in language models, with different methods excelling in offline and online settings.

Findings

Offline self-training improves correctness by 13.9%.

Online self-training improves correctness by 25.9%.

Preference optimization outperforms supervised training in online self-training.

Abstract

Recent language models achieve impressive results in tasks involving complex multistep reasoning, but scaling these capabilities further traditionally requires expensive collection of more annotated data. In this work, we explore the potential of improving models' reasoning capabilities without new data, merely using automated feedback to the validity of their predictions in arithmetic reasoning (self-training). In systematic experimentation across six different arithmetic reasoning datasets, we find that models can substantially improve in both single-round (offline) and online self-training, reaching a correct result in +13.9% and +25.9% more cases, respectively, underlining the importance of actuality of self-training feedback. We further find that in the single-round, offline self-training, traditional supervised training can deliver gains comparable to preference optimization, but in online self-training, preference optimization methods largely outperform supervised training thanks to their superior stability and robustness on unseen types of problems.