Multi-task Code LLMs: Data Mix or Model Merge?

TL;DR

This paper compares data mixing and model merging strategies for creating efficient multi-task code LLMs, finding that model merging performs better at larger scales, while data mixing is preferable at smaller scales.

Contribution

It provides an extensive empirical comparison of data mixing versus model merging for multi-task code LLMs across different scales and introduces a weight analysis technique for understanding task effects.

Findings

Model merging achieves superior performance at larger scales.

Merged models can outperform task-specific fine-tuned models.

Data mixing is preferred at smaller scales.

Abstract

Recent research advocates deploying smaller, specialized code LLMs in agentic frameworks alongside frontier models, sparking interest in efficient strategies for multi-task learning that balance performance, constraints, and costs. We compare two approaches for creating small, multi-task code LLMs: data mixing versus model merging. We conduct extensive experiments across two model families (Qwen Coder and DeepSeek Coder) at two scales (2B and 7B parameters), fine-tuning them for code generation and code summarization tasks. Our evaluation on HumanEval, MBPP, and CodeXGlue benchmarks reveals that model merging achieves the best overall performance at larger scale across model families, retaining 96% of specialized model performance on code generation tasks while maintaining summarization capabilities. Notably, merged models can even surpass individually fine-tuned models, with our best configuration of Qwen Coder 2.5 7B model achieving 92.7% Pass@1 on HumanEval compared to 90.9% for its task-specific fine-tuned equivalent. At a smaller scale we find instead data mixing to be a preferred strategy. We further introduce a weight analysis technique to understand how different tasks affect model parameters and their implications for merging strategies. The results suggest that careful merging and mixing strategies can effectively combine task-specific capabilities without significant performance degradation, making them ideal for resource-constrained deployment scenarios.