LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks
Akshara Prabhakar, Yuanzhi Li, Karthik Narasimhan, Sham Kakade, Eran, Malach, Samy Jelassi

TL;DR
This paper introduces a simple and effective method called concatenation of LoRAs (CAT) for merging skill-specific modules in large language models, significantly improving performance on compositional tasks without retraining.
Contribution
It demonstrates that merging LoRAs through concatenation outperforms existing model- and data-merging techniques in skill composition tasks.
Findings
CAT outperforms existing merging methods by 43% on math problems.
Model merging is more effective than data mixing for skill composition.
Code and data are publicly available for reproducibility.
Abstract
Low-Rank Adaptation (LoRA) is a popular technique for parameter-efficient fine-tuning of Large Language Models (LLMs). We study how different LoRA modules can be merged to achieve skill composition -- testing the performance of the merged model on a target task that involves combining multiple skills, each skill coming from a single LoRA. This setup is favorable when it is difficult to obtain training data for the target task and when it can be decomposed into multiple skills. First, we identify practically occurring use-cases that can be studied under the realm of skill composition, e.g. solving hard math-word problems with code, creating a bot to answer questions on proprietary manuals or about domain-specialized corpora. Our main contribution is to show that concatenation of LoRAs (CAT), which optimally weights LoRAs that were individually trained on different skills, outperforms…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsIntelligent Tutoring Systems and Adaptive Learning · Robotics and Automated Systems · Context-Aware Activity Recognition Systems
