$n$-Musketeers: Reinforcement Learning Shapes Collaboration Among Language Models

TL;DR

This paper introduces a method for integrating multiple specialized language models through their internal states using trainable attention, enabling effective collaboration and reasoning without large monolithic models, and reveals how expert attention evolves during training.

Contribution

It proposes soft hidden-state collaboration for integrating frozen language model experts via attention, providing a new mechanism for structured reasoning and insights into expert utilization patterns.

Findings

Competitive performance with strong RLVR baselines on reasoning tasks

Emergent specialization in expert attention during training

Static preferences explain simpler task performance, dynamic attention for complex tasks

Abstract

Recent progress in reinforcement learning with verifiable rewards (RLVR) shows that small, specialized language models (SLMs) can exhibit structured reasoning without relying on large monolithic LLMs. We introduce soft hidden-state collaboration, where multiple heterogeneous frozen SLM experts are integrated through their internal representations via a trainable attention interface. Experiments on Reasoning Gym and GSM8K show that this latent integration is competitive with strong single-model RLVR baselines. Ablations further reveal a dual mechanism of expert utilization: for simpler arithmetic domains, performance gains can largely be explained by static expert preferences, whereas more challenging settings induce increasingly concentrated and structured expert attention over training, indicating emergent specialization in how the router connects to relevant experts. Overall, hidden-state collaboration provides a compact mechanism for leveraging frozen experts, while offering an observational window into expert utilization patterns and their evolution under RLVR.