Tuning the Right Foundation Models is What you Need for Partial Label Learning

TL;DR

This paper evaluates the use of foundation models in partial label learning, introduces PartialCLIP for efficient fine-tuning, and analyzes their performance and limitations across multiple datasets and scenarios.

Contribution

It provides a comprehensive empirical evaluation of foundation models in PLL and proposes PartialCLIP, a novel fine-tuning framework for improved generalization.

Findings

Foundation models significantly improve PLL performance.

Performance is similar across different PLL approaches with foundation models.

Models are stable across ambiguity levels but sensitive to model selection.

Abstract

Partial label learning (PLL) seeks to train generalizable classifiers from datasets with inexact supervision, a common challenge in real-world applications. Existing studies have developed numerous approaches to progressively refine and recover ground-truth labels by training convolutional neural networks. However, limited attention has been given to foundation models that offer transferrable representations. In this work, we empirically conduct comprehensive evaluations of 11 foundation models across 13 PLL approaches on 8 benchmark datasets under 3 PLL scenarios. We further propose PartialCLIP, an efficient fine-tuning framework for foundation models in PLL. Our findings reveal that current PLL approaches tend to 1) achieve significant performance gains when using foundation models, 2) exhibit remarkably similar performance to each other, 3) maintain stable performance across varying ambiguity levels, while 4) are susceptible to foundation model selection and adaptation strategies. Additionally, we demonstrate the efficacy of text-embedding classifier initialization and effective candidate label filtering using zero-shot CLIP. Our experimental results and analysis underscore the limitations of current PLL approaches and provide valuable insights for developing more generalizable PLL models. The source code can be found at https://github.com/SEU-hk/PartialCLIP.