Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Masks

TL;DR

Model Breadcrumbs is a novel method for merging multiple fine-tunings of foundation models using sparse weight masks, enabling efficient multi-task learning and model updates without extensive hyperparameter tuning.

Contribution

Introduces Model Breadcrumbs, a simple and efficient approach for multi-task model merging using sparse weight masks, avoiding hyperparameter tuning for new tasks.

Findings

Effective in improving multi-task performance

More efficient than previous methods

Does not require hyperparameter tuning for new tasks

Abstract

The rapid development of AI systems has been greatly influenced by the emergence of foundation models. A common approach for targeted problems involves fine-tuning these pre-trained foundation models for specific target tasks, resulting in a rapid spread of models fine-tuned across a diverse array of tasks. This work focuses on the problem of merging multiple fine-tunings of the same foundation model derived from a spectrum of auxiliary tasks. We introduce a new simple method, Model Breadcrumbs, which consists of a sparsely defined weight set that guides model adaptation within the weight space of a pre-trained model. These breadcrumbs are constructed by subtracting the weights from a pre-trained model before and after fine-tuning, followed by a sparsification process that eliminates weight outliers and negligible perturbations. Our experiments demonstrate the effectiveness of Model Breadcrumbs to simultaneously improve performance across multiple tasks. This contribution aligns with the evolving paradigm of updatable machine learning, reminiscent of the collaborative principles underlying open-source software development, fostering a community-driven effort to reliably update machine learning models. Our method is shown to be more efficient and unlike previous proposals does not require hyperparameter tuning for each new task added. Through extensive experimentation involving various models, tasks, and modalities we establish that integrating Model Breadcrumbs offers a simple, efficient, and highly effective approach for constructing multi-task models and facilitating updates to foundation models.