NLPrompt: Noise-Label Prompt Learning for Vision-Language Models

TL;DR

NLPrompt introduces a robust prompt learning framework for vision-language models that effectively handles noisy labels by combining MAE loss and optimal transport-based data purification, leading to improved accuracy and noise resilience.

Contribution

The paper proposes NLPrompt, a novel noise-label prompt learning method that integrates MAE loss and optimal transport for dataset purification, enhancing robustness in vision-language models.

Findings

MAE loss improves robustness against noisy labels in prompt learning.

PromptOT effectively partitions datasets into clean and noisy subsets.

NLPrompt significantly outperforms existing methods across various noise conditions.

Abstract

The emergence of vision-language foundation models, such as CLIP, has revolutionized image-text representation, enabling a broad range of applications via prompt learning. Despite its promise, real-world datasets often contain noisy labels that can degrade prompt learning performance. In this paper, we demonstrate that using mean absolute error (MAE) loss in prompt learning, named PromptMAE, significantly enhances robustness against noisy labels while maintaining high accuracy. Though MAE is straightforward and recognized for its robustness, it is rarely used in noisy-label learning due to its slow convergence and poor performance outside prompt learning scenarios. To elucidate the robustness of PromptMAE, we leverage feature learning theory to show that MAE can suppress the influence of noisy samples, thereby improving the signal-to-noise ratio and enhancing overall robustness. Additionally, we introduce PromptOT, a prompt-based optimal transport data purification method to enhance the robustness further. PromptOT employs text features in vision-language models as prototypes to construct an optimal transportation matrix. This matrix effectively partitions datasets into clean and noisy subsets, allowing for the application of cross-entropy loss to the clean subset and MAE loss to the noisy subset. Our Noise-Label Prompt Learning method, named NLPrompt, offers a simple and efficient approach that leverages the expressive representations and precise alignment capabilities of vision-language models for robust prompt learning. We validate NLPrompt through extensive experiments across various noise settings, demonstrating significant performance improvements.