Black-box Model Merging for Language-Model-as-a-Service with Massive Model Repositories

TL;DR

This paper introduces Evo-Merging, a derivative-free optimization method for black-box model merging of large language models via API calls, effectively combining multiple models without access to their weights.

Contribution

The paper presents a novel black-box model merging framework using evolutionary algorithms, enabling model integration solely through inference API queries for massive LLMs.

Findings

Achieves state-of-the-art results on various tasks

Outperforms existing baselines significantly

Demonstrates effectiveness without access to model weights

Abstract

Model merging refers to the process of integrating multiple distinct models into a unified model that preserves and combines the strengths and capabilities of the individual models. Most existing approaches rely on task vectors to combine models, typically under the assumption that model parameters are accessible. However, for extremely large language models (LLMs) such as GPT-4, which are often provided solely as black-box services through API interfaces (Language-Model-as-a-Service), model weights are not available to end users. This presents a significant challenge, which we refer to as black-box model merging (BMM) with massive LLMs. To address this challenge, we propose a derivative-free optimization framework based on the evolutionary algorithm (Evo-Merging) that enables effective model merging using only inference-time API queries. Our method consists of two key components: (1) sparsity-based denoising, designed to identify and filter out irrelevant or redundant information across models, and (2) sign-aware scaling, which dynamically computes optimal combination weights for the relevant models based on their performance. We also provide a formal justification, along with a theoretical analysis, for our asymmetric sparsification. Extensive experimental evaluations demonstrate that our approach achieves state-of-the-art results on a range of tasks, significantly outperforming existing strong baselines.