Skill Weaving: Efficient LLM Improvement via Modular Skillpacks

TL;DR

SkillWeave is a modular framework that enables large language models to efficiently specialize across multiple domains by partitioning capabilities into lightweight skillpacks, resulting in improved performance and speed.

Contribution

The paper introduces SkillWeave, a novel modular approach with SkillZip compression for multi-domain LLM specialization under fixed memory constraints.

Findings

A 9B SkillWeave model outperforms several baselines.

SkillWeave surpasses a 32B monolithic LLM in performance.

Achieves up to 4x speedup in inference.

Abstract

Large language models increasingly require specialization across diverse domains, yet existing approaches struggle to balance multi-domain capacities with strict memory and inference constraints. In this work, we introduce SkillWeave, a modular improvement framework that enables LLMs to specialize under fixed memory budgets. SkillWeave partitions full capabilities of a general-purpose model into skillpacks -- lightweight, domain-specific delta modules -- that reorganize and refine the model's internal knowledge. For efficient deployment, SkillWeave integrates SkillZip to compress skillpacks into compact and inference-ready format, enabling strong multi-domain performance with low-latency execution. On multi-task and agentic benchmarks, a 9B SkillWeave model outperforms several baselines and even surpasses a 32B monolithic LLM, while achieving up to 4x speedup.