Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts

TL;DR

This paper investigates the effects of fine-tuning on long-tail learning with foundation models, revealing that heavy fine-tuning can harm tail class performance, and proposes a lightweight fine-tuning method called LIFT that improves efficiency and accuracy.

Contribution

The paper uncovers the negative impact of heavy fine-tuning on tail classes and introduces LIFT, a low-complexity, adaptive lightweight fine-tuning algorithm for better long-tail learning.

Findings

Heavy fine-tuning deteriorates tail class performance.

Lightweight fine-tuning outperforms heavy fine-tuning in accuracy.

LIFT reduces training time and model size while maintaining high accuracy.

Abstract

The fine-tuning paradigm in addressing long-tail learning tasks has sparked significant interest since the emergence of foundation models. Nonetheless, how fine-tuning impacts performance in long-tail learning was not explicitly quantified. In this paper, we disclose that heavy fine-tuning may even lead to non-negligible performance deterioration on tail classes, and lightweight fine-tuning is more effective. The reason is attributed to inconsistent class conditions caused by heavy fine-tuning. With the observation above, we develop a low-complexity and accurate long-tail learning algorithms LIFT with the goal of facilitating fast prediction and compact models by adaptive lightweight fine-tuning. Experiments clearly verify that both the training time and the learned parameters are significantly reduced with more accurate predictive performance compared with state-of-the-art approaches. The implementation code is available at https://github.com/shijxcs/LIFT.