Matching Accuracy, Different Geometry: Evolution Strategies vs GRPO in LLM Post-Training

TL;DR

This paper compares Evolution Strategies (ES) and Group Relative Policy Optimization (GRPO) in fine-tuning large language models, revealing they achieve similar accuracy but produce different parameter updates and solutions.

Contribution

It provides a comprehensive analysis and an analytical theory explaining how ES and GRPO differ in solution geometry despite similar task performance.

Findings

ES makes larger, broader parameter changes than GRPO.

ES and GRPO solutions are linearly connected with no loss barrier.

Theoretical framework explains how ES accumulates large off-task movement.

Abstract

Evolution Strategies (ES) have emerged as a scalable gradient-free alternative to reinforcement learning based LLM fine-tuning, but it remains unclear whether comparable task performance implies comparable solutions in parameter space. We compare ES and Group Relative Policy Optimization (GRPO) across four tasks in both single-task and sequential continual-learning settings. ES matches or exceeds GRPO in single-task accuracy and remains competitive sequentially when its iteration budget is controlled. Despite this similarity in task performance, the two methods produce markedly different model updates: ES makes much larger changes and induces broader off-task KL drift, whereas GRPO makes smaller, more localized updates. Strikingly, the ES and GRPO solutions are linearly connected with no loss barrier, even though their update directions are nearly orthogonal. We develop an analytical theory of ES that explains all these phenomena within a unified framework, showing how ES can accumulate large off-task movement on weakly informative directions while still making enough progress on the task to match gradient-based RL in downstream accuracy. These results show that gradient-free and gradient-based fine-tuning can reach similarly accurate yet geometrically distinct solutions, with important consequences for forgetting and knowledge preservation. The source code is publicly available: https://github.com/Bhoy1/ESvsGRPO.