Souper-Model: How Simple Arithmetic Unlocks State-of-the-Art LLM Performance

TL;DR

This paper introduces SoCE, a novel model souping method that uses benchmark-based clustering and weighted averaging of models to significantly improve large language model performance across various domains.

Contribution

The paper presents a new principled approach for model souping that identifies category-specific experts and applies optimized weighted averaging, outperforming previous uniform-averaging methods.

Findings

Achieves state-of-the-art results on the Berkeley Function Calling Leaderboard.

Improves robustness and performance across multilingual, tool calling, and math tasks.

Demonstrates the effectiveness of non-uniform weighted averaging in model souping.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse domains, but their training remains resource- and time-intensive, requiring massive compute power and careful orchestration of training procedures. Model souping-the practice of averaging weights from multiple models of the same architecture-has emerged as a promising pre- and post-training technique that can enhance performance without expensive retraining. In this paper, we introduce Soup Of Category Experts (SoCE), a principled approach for model souping that utilizes benchmark composition to identify optimal model candidates and applies non-uniform weighted averaging to maximize performance. Contrary to previous uniform-averaging approaches, our method leverages the observation that benchmark categories often exhibit low inter-correlations in model performance. SoCE identifies "expert" models for each weakly-correlated category cluster and combines them using optimized weighted averaging rather than uniform weights. We demonstrate that the proposed method improves performance and robustness across multiple domains, including multilingual capabilities, tool calling, and math and achieves state-of-the-art results on the Berkeley Function Calling Leaderboard.