Webly Supervised Image Classification with Self-Contained Confidence

TL;DR

This paper introduces Self-Contained Confidence (SCC), a confidence-based method to improve webly supervised image classification by adaptively balancing pseudo labels and web labels, achieving state-of-the-art results.

Contribution

The paper proposes SCC, a novel confidence measure for WSL, and develops SCC-friendly regularization, notably graph-enhanced mixup, to enhance classification performance.

Findings

Achieved state-of-the-art results on WebVision-1000 and Food101-N datasets.

Demonstrated effectiveness of SCC in balancing pseudo and web labels.

Validated the proposed regularization methods improve model confidence.

Abstract

This paper focuses on webly supervised learning (WSL), where datasets are built by crawling samples from the Internet and directly using search queries as web labels. Although WSL benefits from fast and low-cost data collection, noises in web labels hinder better performance of the image classification model. To alleviate this problem, in recent works, self-label supervised loss $\mathcal{L}_s$ is utilized together with webly supervised loss $\mathcal{L}_w$. $\mathcal{L}_s$ relies on pseudo labels predicted by the model itself. Since the correctness of the web label or pseudo label is usually on a case-by-case basis for each web sample, it is desirable to adjust the balance between $\mathcal{L}_s$ and $\mathcal{L}_w$ on sample level. Inspired by the ability of Deep Neural Networks (DNNs) in confidence prediction, we introduce Self-Contained Confidence (SCC) by adapting model uncertainty for WSL setting, and use it to sample-wisely balance $\mathcal{L}_s$ and $\mathcal{L}_w$. Therefore, a simple yet effective WSL framework is proposed. A series of SCC-friendly regularization approaches are investigated, among which the proposed graph-enhanced mixup is the most effective method to provide high-quality confidence to enhance our framework. The proposed WSL framework has achieved the state-of-the-art results on two large-scale WSL datasets, WebVision-1000 and Food101-N. Code is available at https://github.com/bigvideoresearch/SCC.