To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Models

TL;DR

This paper compares two multi-domain reinforcement learning paradigms for large language models, analyzing their effects on performance across various tasks and revealing insights into their internal mechanisms.

Contribution

It provides a comprehensive comparison and analysis of mixed multi-task RLVR and separate RLVR followed by model merging for large language models across multiple domains.

Findings

RLVR across domains shows minimal mutual interference.

Reasoning-intensive domains have mutually synergistic effects.

Internal mechanisms analyzed include weight space geometry and self-verification.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) plays a key role in stimulating the explicit reasoning capability of Large Language Models (LLMs). We can achieve expert-level performance in some specific domains via RLVR, such as coding or math. When a general multi-domain expert-level model is required, we need to carefully consider the collaboration of RLVR across different domains. The current state-of-the-art models mainly employ two different training paradigms for multi-domain RLVR: mixed multi-task RLVR and separate RLVR followed by model merging. However, most of the works did not provide a detailed comparison and analysis about these paradigms. To this end, we choose multiple commonly used high-level tasks (e.g., math, coding, science, instruction following, and agent) as our target domains and design extensive qualitative and quantitative experiments using open-source datasets. We find the RLVR across domains exhibits few mutual interferences, and reasoning-intensive domains demonstrate mutually synergistic effects. Furthermore, we analyze the internal mechanisms of mutual gains from the perspectives of weight space geometry, information constraints, model prediction behavior and self-verification. This project is named as M2RL that means Mixed multi-task training or separate training followed by model Merging for Reinforcement Learning, and the homepage is at https://github.com/Mosi-AI/M2RL.