Revising the Problem of Partial Labels from the Perspective of CNNs' Robustness

TL;DR

This paper emphasizes the importance of robustness in partial-label CNN training, proposing new attack models and evaluation metrics to assess both accuracy and resilience against adversarial label missing scenarios.

Contribution

It introduces attack models for generating partial-label datasets and employs D-Score for comprehensive robustness evaluation, highlighting limitations of accuracy-only assessments.

Findings

Robustness is often overlooked in partial-label learning evaluations.

Some methods improve accuracy but not robustness, or even reduce it.

Proposed attack models effectively test model resilience against label missing attacks.

Abstract

Convolutional neural networks (CNNs) have gained increasing popularity and versatility in recent decades, finding applications in diverse domains. These remarkable achievements are greatly attributed to the support of extensive datasets with precise labels. However, annotating image datasets is intricate and complex, particularly in the case of multi-label datasets. Hence, the concept of partial-label setting has been proposed to reduce annotation costs, and numerous corresponding solutions have been introduced. The evaluation methods for these existing solutions have been primarily based on accuracy. That is, their performance is assessed by their predictive accuracy on the test set. However, we insist that such an evaluation is insufficient and one-sided. On one hand, since the quality of the test set has not been evaluated, the assessment results are unreliable. On the other hand, the partial-label problem may also be raised by undergoing adversarial attacks. Therefore, incorporating robustness into the evaluation system is crucial. For this purpose, we first propose two attack models to generate multiple partial-label datasets with varying degrees of label missing rates. Subsequently, we introduce a lightweight partial-label solution using pseudo-labeling techniques and a designed loss function. Then, we employ D-Score to analyze both the proposed and existing methods to determine whether they can enhance robustness while improving accuracy. Extensive experimental results demonstrate that while certain methods may improve accuracy, the enhancement in robustness is not significant, and in some cases, it even diminishes.