FusionBench: A Unified Library and Comprehensive Benchmark for Deep Model Fusion

TL;DR

FusionBench is a comprehensive, open-source benchmark and library that standardizes evaluation and facilitates development of deep model fusion techniques across diverse tasks and model configurations.

Contribution

It introduces the first unified benchmark and library dedicated to deep model fusion, enabling consistent evaluation and easier implementation of new methods.

Findings

Provides a diverse set of tasks and datasets for evaluation.

Facilitates comparison of fusion methods across scenarios.

Supports easy implementation and testing of new fusion techniques.

Abstract

Deep model fusion is an emerging technique that unifies the predictions or parameters of several deep neural networks into a single better-performing model in a cost-effective and data-efficient manner. Although a variety of deep model fusion techniques have been introduced, their evaluations tend to be inconsistent and often inadequate to validate their effectiveness and robustness. We present FusionBench, the first benchmark and a unified library designed specifically for deep model fusion. Our benchmark consists of multiple tasks, each with different settings of models and datasets. This variety allows us to compare fusion methods across different scenarios and model scales. Additionally, FusionBench serves as a unified library for easy implementation and testing of new fusion techniques. FusionBench is open source and actively maintained, with community contributions encouraged. Homepage https://github.com/tanganke/fusion_bench