FoPro: Few-Shot Guided Robust Webly-Supervised Prototypical Learning

TL;DR

FoPro is a novel few-shot guided prototypical learning method that leverages limited real-world data to improve webly supervised learning performance in real-world domains, achieving state-of-the-art results.

Contribution

The paper introduces FoPro, a method that combines few-shot real-world data with web data using contrastive learning and prototype refinement to enhance real-world domain performance.

Findings

Achieves state-of-the-art results on multiple datasets.

Outperforms existing WSL methods in few-shot settings.

Effectively narrows the domain gap between web and real-world data.

Abstract

Recently, webly supervised learning (WSL) has been studied to leverage numerous and accessible data from the Internet. Most existing methods focus on learning noise-robust models from web images while neglecting the performance drop caused by the differences between web domain and real-world domain. However, only by tackling the performance gap above can we fully exploit the practical value of web datasets. To this end, we propose a Few-shot guided Prototypical (FoPro) representation learning method, which only needs a few labeled examples from reality and can significantly improve the performance in the real-world domain. Specifically, we initialize each class center with few-shot real-world data as the ``realistic" prototype. Then, the intra-class distance between web instances and ``realistic" prototypes is narrowed by contrastive learning. Finally, we measure image-prototype distance with a learnable metric. Prototypes are polished by adjacent high-quality web images and involved in removing distant out-of-distribution samples. In experiments, FoPro is trained on web datasets with a few real-world examples guided and evaluated on real-world datasets. Our method achieves the state-of-the-art performance on three fine-grained datasets and two large-scale datasets. Compared with existing WSL methods under the same few-shot settings, FoPro still excels in real-world generalization. Code is available at https://github.com/yuleiqin/fopro.