KALAVAI: Predicting When Independent Specialist Fusion Works -- A Quantitative Model for Post-Hoc Cooperative LLM Training

TL;DR

KALAVAI introduces a predictive model for post-hoc fusion of independent domain specialists into a single model, demonstrating consistent gains and effective routing with minimal divergence.

Contribution

The paper presents a protocol enabling predictable, high-performance fusion of independently trained specialists using lightweight routing and shared initialization.

Findings

Fusion gains are predictable based on divergence, with a strong correlation (R^2=0.856).

Lightweight MoE routing achieves near-oracle domain assignment accuracy (<10^{-5} nats).

Cross-lingual fusion yields significant perplexity reductions, e.g., Yoruba from 41.9 to 7.7.

Abstract

Independently trained domain specialists can be fused post-hoc into a single model that outperforms any individual specialist, and the gain is predictable: gain = 0.82 x divergence - 2.72 (R^2 = 0.856, n=6, 3-26% divergence). This enables practitioners to estimate cooperative value before committing compute. Below ~3.3% divergence, gains approach zero.In the KALAVAI protocol, contributors fine-tune copies of a shared checkpoint independently, then submit for lightweight MoE routing (500 steps). Gains are consistent: +7.72% at 410M (+/-0.02%, 3 seeds), +7.49% at 1B (+/-0.01%, 3 seeds), +6.53% at 6.9B, each over the best specialist. The router matches domain-oracle routing within <10^{-5} nats. Cross-lingual fusion (Tamil/Yoruba/Welsh/Code) achieves +21.76%, with Yoruba perplexity falling 41.9 to 7.7. A 20-contributor federation achieves +16.71% (+/-0.07pp, 3 seeds).Three requirements bound the protocol. Shared initialisation is necessary: checkpoint mismatch degrades routing. Frozen layers are optional below ~10,000 steps and beneficial beyond. Learned routing is essential: uniform averaging degrades by -1.2% vs. best specialist, while any trained router achieves oracle-optimal assignment.