Evolutionary Strategies lead to Catastrophic Forgetting in LLMs

TL;DR

This paper investigates how Evolutionary Strategies (ES), a gradient-free learning method, cause significant forgetting in large language models during continual training, highlighting a key challenge for online learning.

Contribution

The study provides a comprehensive analysis of ES's forgetting behavior in LLMs, revealing its limitations for continual learning and comparing it to gradient-based methods like GRPO.

Findings

ES achieves near-GRPO performance on math and reasoning tasks

ES exhibits significant forgetting of prior abilities during training

ES updates are less sparse and have larger $ extit{l}_2$ norms than GRPO updates

Abstract

One of the biggest missing capabilities in current AI systems is the ability to learn continuously after deployment. Implementing such continually learning systems have several challenges, one of which is the large memory requirement of gradient-based algorithms that are used to train state-of-the-art LLMs. Evolutionary Strategies (ES) have recently re-emerged as a gradient-free alternative to traditional learning algorithms and have shown encouraging performance on specific tasks in LLMs. In this paper, we perform a comprehensive analysis of ES and specifically evaluate its forgetting curves when training for an increasing number of update steps. We first find that ES is able to reach performance numbers close to GRPO for math and reasoning tasks with a comparable compute budget. However, and most importantly for continual learning, the performance gains in ES is accompanied by significant forgetting of prior abilities, limiting its applicability for training models online. We also explore the reason behind this behavior and show that the updates made using ES are much less sparse and have orders of magnitude larger $\ell_2$ norm compared to corresponding GRPO updates, explaining the contrasting forgetting curves between the two algorithms. With this study, we aim to highlight the issue of forgetting in gradient-free algorithms like ES and hope to inspire future work to mitigate these issues.