LT-Soups: Bridging Head and Tail Classes via Subsampled Model Soups

TL;DR

This paper introduces LT-Soups, a two-stage model averaging method that improves performance across long-tailed distributions by balancing head and tail class accuracy, outperforming existing PEFT methods.

Contribution

The paper proposes LT-Soups, a novel two-stage model soup technique that enhances long-tailed distribution handling by reducing head-class bias and restoring head-class accuracy.

Findings

LT-Soups outperforms PEFT in diverse LT regimes.

The method achieves better head-tail trade-offs.

Experiments on six datasets validate its effectiveness.

Abstract

Real-world datasets typically exhibit long-tailed (LT) distributions, where a few head classes dominate and many tail classes are severely underrepresented. While recent work shows that parameter-efficient fine-tuning (PEFT) methods like LoRA and AdaptFormer preserve tail-class performance on foundation models such as CLIP, we find that they do so at the cost of head-class accuracy. We identify the head-tail ratio, the proportion of head to tail classes, as a crucial but overlooked factor influencing this trade-off. Through controlled experiments on CIFAR100 with varying imbalance ratio ($\rho$) and head-tail ratio ($\eta$), we show that PEFT excels in tail-heavy scenarios but degrades in more balanced and head-heavy distributions. To overcome these limitations, we propose LT-Soups, a two-stage model soups framework designed to generalize across diverse LT regimes. In the first stage, LT-Soups averages models fine-tuned on balanced subsets to reduce head-class bias; in the second, it fine-tunes only the classifier on the full dataset to restore head-class accuracy. Experiments across six benchmark datasets show that LT-Soups achieves superior trade-offs compared to both PEFT and traditional model soups across a wide range of imbalance regimes.