More Reliable Pseudo-labels, Better Performance: A Generalized Approach to Single Positive Multi-label Learning

TL;DR

This paper introduces a new framework for single positive multi-label learning that improves label reliability and model performance by using a robust loss function and dynamic pseudo-labeling strategies, achieving state-of-the-art results.

Contribution

It proposes the GPR Loss and DAMP techniques within the AEVLP framework to enhance learning from partially annotated multi-label data.

Findings

Significant performance improvements on four benchmark datasets.

State-of-the-art results in multi-label classification.

Effective handling of noisy pseudo-labels.

Abstract

Multi-label learning is a challenging computer vision task that requires assigning multiple categories to each image. However, fully annotating large-scale datasets is often impractical due to high costs and effort, motivating the study of learning from partially annotated data. In the extreme case of Single Positive Multi-Label Learning (SPML), each image is provided with only one positive label, while all other labels remain unannotated. Traditional SPML methods that treat missing labels as unknown or negative tend to yield inaccuracies and false negatives, and integrating various pseudo-labeling strategies can introduce additional noise. To address these challenges, we propose the Generalized Pseudo-Label Robust Loss (GPR Loss), a novel loss function that effectively learns from diverse pseudo-labels while mitigating noise. Complementing this, we introduce a simple yet effective Dynamic Augmented Multi-focus Pseudo-labeling (DAMP) technique. Together, these contributions form the Adaptive and Efficient Vision-Language Pseudo-Labeling (AEVLP) framework. Extensive experiments on four benchmark datasets demonstrate that our framework significantly advances multi-label classification, achieving state-of-the-art results.

Tables

Table 1. Table 1 : Experimental results on various benchmarks with SPML setting according to [ 5 ] . Our results are averaged over three runs as suggested in [ 14 ] .

Method	VOC	COCO	NUS	CUB
Full-label (BCE)	89.42	76.78	52.08	30.90
Assume Negative (CVPR’21) [5]	85.89	64.92	42.27	18.31
ROLE (CVPR’21) [5]	87.77	67.04	41.63	13.66
EM (ECCV’22) [34]	89.09	70.70	47.15	20.85
EM + APL (ECCV’22) [34]	89.19	70.87	47.59	21.84
BoostLU + LL-R (CVPR’23) [14]	89.29	72.89	49.59	19.8
SMILE (NeurIPS’22) [28]	86.31	63.33	43.61	18.61
MIME (ICML’23) [19]	89.20	72.92	48.74	21.89
GR Loss (IJCAI’24) [2]	89.83	73.17	49.08	21.64
VLPL (CVPRW’24) [27]	89.10	71.45	49.55	24.02
AEVLP (Ours)	90.46	73.54	50.70	24.89

Table 2. Table 2 : Comparison of mAP scores when applying GPR Loss to different pseudo-labeling strategies.

Method	VOC	COCO	NUSWIDE	CUB	Average
BCE + Random	86.84	67.46	41.23	10.69	51.56
GPR + Random	87.33	70.00	43.60	14.98	53.98 (+ 2.42)
VLPL	89.1	71.45	49.55	24.02	58.53
GPR + VLPL	89.52	72.83	49.93	24.22	59.13 (+ 0.6)
LL-Ct	89.00	70.50	48.00	20.40	56.98
GPR + LL-Ct	89.5	71.66	47.72	20.41	57.32 (+ 0.34)
BCE + DAMP	88.72	71.89	48.60	24.01	58.31
GPR + DAMP	90.46	73.54	50.7	24.89	59.90 (+ 1.59)

Table 3. Table 3 : DAMP performance on several benchmark datasets.

Metrics	VOC	COCO	NUS	CUB
Average Precision	65.13	84.79	37.09	19.07
Accumulated Precision	60.33	81.59	34.49	18.29
Average Recall	48.29	24.68	8.84	11.37
Accumulated Recall	51.1	26.84	10.06	13.88

Table 4. Table 4 : Ablation study on the main components of the AEVLP framework: noise addition G ​ ( ⋅ ) G(\cdot) , augmentation T ⁡ ( ⋅ ) \operatorname{T}(\cdot) , overlapping ratio r r , loss re-weighting for positive pseudo-labels v 4 ​ ( p ; α ) v^{4}(p;\alpha) , regularization term ℛ \mathcal{R} , and negative pseudo-labels loss ℒ n , i 3 \mathcal{L}^{3}_{n,i} . Note that ✗ for v 4 ​ ( p ; α ) v^{4}(p;\alpha) indicates it is set to a constant, and for r r , it indicates r = 0 r=0 .

AEVLP						Datasets
DAMP			GPR Loss			VOC	COCO	NUS	CUB
$G(\cdot )$	$T(\cdot )$	$r$	$v^4(p;\alpha )$	$ℛ$	${ℒ}_{n,i}^3$
✗	✓	✗	✗	✗	✗	90.28	73.07	50.00	24.14
✓	✓	✗	✗	✗	✗	90.21	73.37	50.32	24.27
✓	✗	✓	✗	✗	✗	90.13	73.32	50.46	24.15
✓	✓	✓	✗	✗	✗	90.22	73.14	50.57	24.24
✓	✓	✓	✓	✗	✗	90.31	73.40	50.37	24.73
✓	✓	✓	✓	✓	✗	90.38	73.51	50.64	24.83
✓	✓	✓	✓	✓	✓	90.46	73.54	50.70	24.89

Table 5. Table 5 : mAP scores for various grid sizes g g on VOC and CUB datasets.

Grid size	2	3	4	5	6
VOC	90.13	90.08	90.46	90.21	90.21
COCO	73.07	73.13	73.54	73.48	73.32
NUS	50.25	50.19	50.7	50.35	50.15
CUB	24.37	24.38	24.48	24.89	24.67

Table 6. Table 6 : AEVLP performance on various backbone architectures. Note that 1K and 22K refer to pretrained models on ImageNet-1K and ImageNet-22K, respectively.

Backbone	ResNet-50	ConvNeXt-L-1K	ConvNeXt-L-22K	ViT-L/14
VOC	90.46	92.92	93.11	93.10
COCO	73.54	76.14	82.69	81.39
NUS	50.7	50.9	54.74	54.98
CUB	24.89	25.26	24.87	23.36

Equations

$$\text{centering}\mathcal{L}=-\frac{1}{NC}\sum _{n=1}^N\sum _{i=1}^C{y}_{n,i}\log p_{n,i}+(1-y_{n,i})\log (1-p_{n,i})\text{@add@centering}$$

$$\text{centering}{\mathcal{L}}^{GR}=\frac{1}{NC}\sum _{n=1}^N\sum _{i=1}^C{v}^{old}(p_{n,i};\alpha )\cdot {\mathcal{L}}_{n,i}^{old}\text{@add@centering}$$

$$v^{old}(p_{n,i};\alpha )=\text{mathds}{1}_{[{\hat{y}}_{n,i}=1]}{v}^1(p_{n,i};\alpha )+\text{mathds}{1}_{[{\hat{y}}_{n,i}=0]}{v}^2(p_{n,i};\alpha ),$$

$${\mathcal{L}}_{n,i}^{old}=\text{mathds}{1}_{[{\hat{y}}_{n,i}=1]}{\mathcal{L}}_{n,i}^1+\text{mathds}{1}_{[{\hat{y}}_{n,i}=0]}{\mathcal{L}}_{n,i}^2,$$

$${\mathcal{L}}_{n,i}^2=(1-\hat{k}(p_{n,i};\beta ))\frac{1-(1-p_{n,i}{)}^{q_1}}{q_1}+\hat{k}(p_{n,i};\beta )\frac{1-p_{n,i}^{q_2}}{q_2}.$$

$$\text{centering}{\mathcal{L}}^{GPR}=\frac{1}{NC}\sum _{n=1}^N\sum _{i=1}^C{v}^{new}(p_{n,i};\alpha )\cdot {\mathcal{L}}_{n,i}^{new}+\eta \mathcal{R}.\text{@add@centering}$$

$${\mathcal{L}}_{n,i}^{new}=$$

$$+\text{mathds}{1}_{[{\hat{y}}_{n,i}=0\wedge l_{n,i}=-1]}{\mathcal{L}}_{n,i}^3+\text{mathds}{1}_{[{\hat{y}}_{n,i}=0\wedge l_{n,i}=1]}{\mathcal{L}}_{n,i}^4,$$

$$v^{new}(p_{n,i};\alpha )=$$

$$+\text{mathds}{1}_{[{\hat{y}}_{n,i}=0\wedge l_{n,i}=0]}{v}^2(p_{n,i};\alpha )$$

$$+\text{mathds}{1}_{[{\hat{y}}_{n,i}=0\wedge l_{n,i}=-1]}{v}^3(p_{n,i};\alpha )$$

$$+\text{mathds}{1}_{[{\hat{y}}_{n,i}=0\wedge l_{n,i}=1]}{v}^4(p_{n,i};\alpha ).$$

$${\mathcal{L}}_{n,i}^3$$

$${\mathcal{L}}_{n,i}^4$$

$$v^{4}(p;\alpha)=\min\Biggl{(}\max\Bigl{(}1-\exp\left(-\frac{(p-\mu)^{2}}{2\sigma^{2}}\right),\lambda_{1}\Bigr{)},\lambda_{2}\Biggr{)}$$

$$\mathcal{R}={\left(\frac{\hat{m}-m}{C}\right)}^2,$$

$$m\approx E_{pos}={\mathbf{E}}_{(x,y)\sim p_{\text{data }}(x,y)}\sum _{i=1}^C\text{mathds}{1}_{\left[y_i=1\right]},$$

$$\hat{m}=\frac{1}{N}\sum _{n=1}^N\sum _{i=1}^C{p}_{n,i}.$$

$$\mathbf{C}(\mathcal{M})=\exp \left(\sum _{n=1}^N\sum _{i=1}^C\text{mathds}{1}_{[{\hat{y}}_{n,i}=1]}\log P(l_{n,i}=1\mid x_n,\mathcal{M})\right),$$

$$m^{\prime }=\frac{1}{N^{\prime }}\sum _{n=1}^{N^{\prime }}\sum _{i=1}^C\text{mathds}{1}_{[y_{n,i}^{val}=1]}.$$

$$l_c^{\prime }=\{\begin{array}{ll}1,& s_c^{final}\in \mathrm{TopK}(S^{final},k)\&s_c^{final}\ge {\zeta }^{global}\\ 0,& \text{otherwise},\end{array}$$

$$S^{avg}=\frac{1}{2}\left(S^{global}+\frac{1}{R}\sum _{z=1}^R{S}_z^{local}\right).$$

$$l_c=\{\begin{array}{ll}-1,& s_c^{\text{avg}}\le {\theta }_{{\Delta }_{\text{neg}}}(S^{\text{avg}})\\ l_c^{\prime },& \text{otherwise}\end{array}$$

Full text

1 Introduction

Most visual classification studies focus on multi-class settings which aim to identify the most suitable label for a given input image from a set of possible options. However, in real-world scenarios, an image often contains multiple objects or attributes that can be considered as their labels. Therefore, multi-label learning [20, 31] has emerged as a solution to develop predictive models capable of assigning multiple labels to an unseen image. In early stages, multi-label learning required annotation of all relevant classes for each training instance; however, such an exhaustive annotation scheme is often costly and impractical [7].

Due to the cost and impracticality of exhaustive annotation [7], researchers have explored multi-label learning with missing labels [25, 29, 26, 33]. In its extreme setting, known as Single-Positive Multi-Label Learning (SPML) [5], each image is annotated with only one confirmed positive label, leaving the remaining labels unknown. This approach significantly reduces labeling cost and demonstrates that multi-label classifiers can be learned with minimal supervision.

A common strategy in SPML is to treat missing labels as negative (the Assume Negative method [5]), and many existing methods [2, 28, 34, 19] further rely on the model’s internal knowledge to generate pseudo-labels for fully supervised learning. However, this approach often produces inherently noisy pseudo-labels, leading to repeated errors during training, an issue that prior studies have not effectively addressed. To tackle this challenge, we propose the Generalized Pseudo-label Robust Loss (GPR Loss), a novel loss function designed to learn effectively from pseudo-labels generated by an external knowledge-based method while robustly mitigating the noise associated with uncertain labels.

Recently, the emergence of Vision-Language Models (VLMs) has enhanced performance in various recognition tasks [30], including multi-label learning. One approach involves fine-tuning these models with additional network layers [8], while another leverages model outputs with pseudo-labels in an unsupervised, annotation-free manner [1]. Recent work [27] generates a fixed pseudo-label vector for each image using a pretrained large CLIP model. However, relying on a single fixed pseudo-label vector prevents the model from re-evaluating and correctly associating an image with categories that were initially misclassified as false negatives during training.

Based on these observations, to verify the strength of GPR Loss, we introduce a simple yet effective dynamic pseudo-labeling process that encourages CLIP to attend not only to the global image but also to various local areas, which are randomly selected and augmented during training. This simple pseudo-labeling approach allows the pseudo-label vector for each image to vary from epoch to epoch, reducing the chance of missing correct associations.

In summary, the main contributions of this work are twofold. First, we propose the Generalized Pseudo-label Robust Loss (GPR Loss) to effectively learn from pseudo-labels generated by an external knowledge-based method while mitigating the noise from uncertain labels. Second, we introduce the Dynamic Augmented Multi-focus Pseudo-labeling (DAMP) technique to generate reliable pseudo-labels in a stable manner, which, together with GPR Loss, forms the AEVLP framework. Through extensive experiments, to the best of our knowledge, we demonstrate that this framework achieves state-of-the-art (SOTA) results for the SPML problem compared to existing methods based on loss design and pseudo-labeling.

2 Related Works

In SPML, two strategies are employed to handle missing labels. The first strategy treats missing labels as unknown variables to be predicted [23]. The second strategy assumes that all missing labels are negative, transforming SPML into a fully supervised multi-label learning problem [5, 28, 34, 14, 13]. The Assume Negative assumption remains popular in multi-label learning with missing labels including SPML [19] and often serves as a baseline in benchmark experiments.

However, this naive assumption oversimplifies the problem and can lead to numerous false negative labels [32]. To address this issue, Cole et al. [5] propose the online estimation of labels during training. Furthermore, Kim et al. [13] show that reducing large losses, which are potentially introduced by false negatives, can significantly improve performance. Meanwhile, the authors in [14] modify the final stage of the CNN architecture to increase the attribution scores of output logits. In [34], entropy maximization (EM) and asymmetric pseudo-labeling (APL) are combined to improve robustness. Additionally, Chen et al. [2] adapt class- and instance-wise loss concepts to SPML within a comprehensive framework. In [28, 19], the approach introduces a label enhancement process, enhancing model performance with minimal positive labels per instance. While our study continues to rely on the Assume Negative assumption and addresses the resulting false negatives, we also focus on learning from pseudo-labels and mitigating the noise they may introduce.

To address the challenges caused by incomplete labeling in multi-label learning, the ability of the CLIP model [22] for zero-shot classification has emerged as an effective solution. Recent work by [27] forces the model to learn from pseudo-labels generated by CLIP. Another approach [8] finetunes CLIP model with additional layers of a Graph Convolutional Network (GCN) [16]. Additionally, in [1], the approach is to generate soft pseudo-labels based on global-local image-text similarity aggregation. Our approach also leverages CLIP for pseudo-labeling; however, unlike previous methods, it employs dynamic pseudo-labeling throughout the training process, reassigning each image a new set of pseudo-labels at every epoch. This helps mitigate pseudo-labeling errors while expanding the range of correct pseudo-labels.

3 Problem Statement

In the original multi-label learning problem, given a dataset $\mathcal{D}=\{(x_{n},y_{n})\}_{n=1}^{N}$, where $x_{n}\in\mathcal{X}$ (input space) and $y_{n}\in\mathcal{Y}=\{0,1\}^{C}$ (label space), and a validation dataset $\mathcal{D}^{\text{val}}=\{(x^{\text{val}}_{n},y^{\text{val}}_{n})\}_{n=1}^{N^{\prime}}$, the goal is to learn a mapping $f:\mathcal{X}\rightarrow\mathcal{Y}$ that predicts multiple labels for each $x_{n}$. We denote $p_{n}=f(x_{n})=\sigma(s_{n})$, where $s_{n}$ represents the model’s logits and $\sigma$ is the sigmoid activation function. The loss function is defined as follows:

[TABLE]

To address the SPML problem, we apply the Assume Negative assumption [5], which treats unobserved labels as negative, transforming the problem into the original multi-label learning setup. The original dataset $\mathcal{D}$ is replaced with a pseudo-multi-label dataset $\hat{\mathcal{D}}=\{(x_{n},\hat{y}_{n})\}_{n=1}^{N}$, where $\hat{y}_{n}\in\hat{\mathcal{Y}}=\{0,1\}^{C}$, ensuring $\sum_{i=1}^{C}\hat{y}_{n,i}=1$. Each $\hat{y}_{n,i}=1$ if $x_{n}$ is relevant to class $i$ (positive label) and 0 if the label is unknown or unannotated.

In SPML, based on pseudo-labeling, an instance can be represented as a triplet $(x_{n},\hat{y}_{n},l_{n})$, where $x_{n}$ is the input, $\hat{y}_{n}$ is the label vector based on the Assume Negative assumption, and $l_{n}\in\{-1,0,1\}^{C}$ represents pseudo-labels generated by an arbitrary method $\mathcal{M}$ to approximate additional labels. For class $i$, $l_{n,i}=1$ indicates a positive pseudo-label, $l_{n,i}=-1$ a negative pseudo-label, and $l_{n,i}=0$ means the label is undefined by $\mathcal{M}$.

4 Generalized Pseudo-Label Robust Loss

4.1 Background

To address the noise introduced in the Assume Negative assumption, the authors in [2] propose the Generalized Robust Loss (GR Loss) as follows:

[TABLE]

where the probability of class $i$ and instance $n$ is denoted as $p_{n,i}$; $v^{old}(p_{n,i};\alpha)$ and $\mathcal{L}_{n,i}^{old}$ are the class-and-instance-specific weight and loss, respectively. Let $\mathds{1}_{[\cdot]}$ is the indicator function, the weight term $v^{old}(p_{n,i};\alpha)$ is expressed as:

[TABLE]

where $v^{1}(p;\alpha)=1$ and $v^{2}(p;\alpha)=\exp\left(-\frac{(p-\mu)^{2}}{2\sigma^{2}}\right)$. Here, $\alpha=[\sigma,\mu]$ is updated linearly over training epochs, following [2], as detailed in the supplementary materials. The loss term $\mathcal{L}^{old}_{n,i}$ is defined as:

[TABLE]

where $\mathcal{L}^{1}_{n,i}=-\log p_{n,i}$ and the term $\mathcal{L}^{2}_{n,i}$ is computed as:

[TABLE]

Here, $\hat{k}(p_{n,i};\beta)$ is the estimate of the label being a false negative during training with a parameter $\beta$, while $q_{1}$ and $q_{2}$, treated as hyperparameters, balance the trade-off between MAE and BCE loss, as described in [2] and illustrated in the supplementary materials. Note that both $\hat{k}(p_{n,i};\beta)$ and $v(p_{n,i};\alpha)$ are implemented with gradients stopped with respect to $p_{n,i}$.

4.2 Proposed Method

The above method, GR Loss in Sec. 4.1, only works with the naive Assume Negative assumption and cannot handle external pseudo-labels generated by a reliable method $\mathcal{M}$. Inspired by [27, 2], we propose the Generalized Pseudo-Label Robust Loss (GPR Loss), as follows:

[TABLE]

We explain the terms in Eq. 2 below. The loss term $\mathcal{L}^{new}_{n,i}$ and the weight term $v^{new}(p_{n,i};\alpha)$ are expressed as follows:

[TABLE]

[TABLE]

The new loss terms $\mathcal{L}^{3}_{n,i}$ and $\mathcal{L}^{4}_{n,i}$ are given by:

[TABLE]

To provide a clearer understanding of our loss function’s design, we outline how it handles various cases based on label status as follows.

Confirmed Positives ($\hat{y}_{n,i}=1$): The loss term $\mathcal{L}^{1}_{n,i}$ is retained to ensure that the model assigns high probability to the true positive classes.

Undefined Pseudo-labels ($\hat{y}_{n,i}=0\land l_{n,i}=0$): The term $\mathcal{L}^{2}_{n,i}$ (inherited from GR Loss) is used when no additional information from pseudo labeling is provided, and the label is naively assumed to be negative due to the high negative rate.

Negative Pseudo-labels ($\hat{y}_{n,i}=0\land l_{n,i}=-1$): The loss term $\mathcal{L}^{3}_{n,i}$, unlike $\mathcal{L}^{2}_{n,i}$ which includes the control of $\hat{k}(p_{n,i})$, directly penalizes the model for assigning a high probability to classes that are considered negative by the pseudo-labels. Moreover, by employing the same weight function $v^{3}(p;\alpha)=\exp\left(-\frac{(p-\mu)^{2}}{2\sigma^{2}}\right),$ which is also used in $\mathcal{L}^{2}_{n,i}$ for undefined pseudo-labels, the method balances the contributions between positive and negative cases.

Positive Pseudo-labels ($\hat{y}_{n,i}=0\land l_{n,i}=1$): The loss term $\mathcal{L}^{4}_{n,i}$ is a softened version of $\mathcal{L}^{1}_{n,i}$ through the coefficient $q_{3}$. This label smoothing helps to temper the model’s confidence, reducing overfitting to noisy pseudo-labels while still promoting the learning of positive associations when a pseudo-label is provided. The weight function

[TABLE]

constrains the influence of this term within a desirable range, controlled by $\lambda_{1}$ and $\lambda_{2}$.

For stable learning when introducing additional pseudo-labels, a regularization term $\mathcal{R}$ is used to restrict the number of positive predictions with a coefficient $\eta$, as in [5]:

[TABLE]

where $m$ is the expected number of positive labels per image that can either be estimated from the available data or treated as a hyperparameter, such as:

[TABLE]

and $\hat{m}$ represents the average sum of probabilities per image, computed as:

[TABLE]

Our loss function $\mathcal{L}^{GPR}$ is a generalization of $\mathcal{L}^{GR}$ as shown in the following theorem.

Theorem 4.1** (Our GPR Loss generalizes GR Loss).**

Given a training dataset $\{(x_{n},\hat{y}_{n},l_{n})\}_{n=1}^{N}$ and a validation set $\{(x^{val}_{n},y^{val}_{n})\}_{n=1}^{N^{\prime}}$ as described in Fig. 1, define:

[TABLE]

and:

[TABLE]

Then $\mathcal{L}^{\text{GPR}}$ tends to $\mathcal{L}^{\text{GR}}$ when $\max\left(\mathbf{C}(\mathcal{M}),\left|m^{\prime}-\hat{m}\right|\right)$ tends to zero. Here, $\hat{m}$ is defined in Eq. 6.

Remark 1**.**

Intuitively, the above theorem says that our loss function $\mathcal{L}^{\text{GPR}}$ will become the loss function $\mathcal{L}^{\text{GR}}$ when $\max\left(\mathbf{C}(\mathcal{M}),\left|m^{\prime}-\hat{m}\right|\right)$ tends to zero. The quantity $\mathbf{C}(\mathcal{M})$ represents the confidence of method $\mathcal{M}$ within the range $(0,e]$ when generating pseudo-labels based on the given single positive label ($\hat{y}_{n,i}=1$). Lower values of $\mathbf{C}(\mathcal{M})$, particularly those below $\epsilon$, indicate a significant divergence between the pseudo-label probability distribution and the ground truth distribution, leading to $l_{n,i}=0$ for all $n,i$.

Remark 2**.**

When $N^{\prime}$ is sufficiently large, $m^{\prime}\to E_{pos}$, with $E_{pos}$ defined in Eq. 5, then $\hat{m}\to E_{pos}$. As a result, the regularization term in Eq. 2 vanishes, i.e., $\mathcal{R}\to 0$. This demonstrates that the GPR Loss naturally calibrates itself when the pseudo-label confidence and the expected positive label statistics align.

The proof of this theorem can be found in the supplementary materials.

5 DAMP

In this section, we introduce a simple yet effective pseudo-labeling method that integrates with GPR Loss via a Dynamic Augmented Multi-focus Pseudo-labeling (DAMP) approach to form the framework AEVLP, as shown in Fig. 1, for SPML. Further implementation details are provided in the supplementary materials.

5.1 CLIP Inference and Strengthening with Noise

As introduced in [22], given an image input $x$ and the $i$-th class from a set of $C$ classes, the corresponding visual embedding and textual embedding are $h=E_{v}(x)\in\mathbb{R}^{K}$ and $t_{i}=E_{t}(\mathcal{P}_{i})\in\mathbb{R}^{K}$, respectively. Here, $E_{v}$ and $E_{t}$ are the image and text encoders of the CLIP model with dimension $K$, and $\mathcal{P}_{i}$ is a predefined prompt for class $i$, such as “a photo of a {class}”. Let $\hat{s}_{i}=\operatorname{Cos}(h,t_{i})$, $s_{i}=\operatorname{Softmax}\left(\frac{\hat{s}_{i}}{\tau}\right)$, and denote $S=\{s_{1},s_{2},\ldots,s_{C}\}$ as the probability distribution for $x$ across $C$ classes.

Several works, including [35, 17, 12], have studied enhancing model performance during fine-tuning by adding controlled noise to model embeddings. Additionally, in [8, 3], label-to-label relationships are presented by GCN. Inspired by this, we propose adding controlled label-to-label correspondence noise to the text embeddings of the CLIP model. Concretely, we redefine the text embedding and update the GCN as: $t_{i}=G(E_{t}(\mathcal{P}_{i}))+E_{t}(\mathcal{P}_{i})$ and $H_{l+1}=\operatorname{LeakyReLU}(A^{*}H_{l}W_{l})$ for $l\in[0,L)$, with $H_{0}=\{E_{t}(\mathcal{P}_{i})\mid 1\leq i\leq C\}$. The graph $G$ remains frozen during training, and the weights $W_{l}$ are initialized from a uniform distribution. The adjacency matrix $A^{*}$ is derived from the cosine similarity scores between the text embeddings of the classes produced by the CLIP text encoder.

5.2 Dynamic Augmented Multi-focus Pseudo-labeling

Augmentor.

Let $I$ be an image. It is partitioned into overlapping patches $\{P_{z}\}_{z=1}^{R}$ by dividing it into a grid and slightly enlarging each patch by a random ratio to ensure overlap. We then process the image and its patches using a transformation pipeline $\operatorname{T}(\cdot)$, which includes standard preprocessing and weak data augmentation techniques, to generate various views for CLIP: the global view $x^{global}=\operatorname{T}(I)$ and the local views $x^{local}_{z}=\operatorname{T}(P_{z})$. Following Sec. 5.1, these views yield the probability distributions $S^{global}=\{s^{global}_{1},s^{global}_{2},\ldots,s^{global}_{C}\}$ and $S^{local}_{z}=\{s^{local}_{z,1},s^{local}_{z,2},\ldots,s^{local}_{z,C}\}$.

Local threshold based on single positives.

Let $\hat{c}$ be the given single positive label, according to the SPML setting in Fig. 1, for the image $I$. The local threshold $\zeta^{local}$, which defines the patches to be trusted, is adjusted based on $\zeta^{local}=\min(s_{\hat{c}}^{global},\nu)$, where $\nu$ is the general local threshold, set as a hyperparameter. In some cases, if $\nu$ is set too high to recognize hard positives, we should consider the scores above $s_{\hat{c}}^{global}$, as these can be meaningful, since $\hat{c}$ is one of the true labels of the global view.

Aggregator.

From the local distributions $\{S_{z}^{\text{local}}\}_{z=1}^{R}$, we aggregate a unified local distribution $S^{\text{agg}}$ following [1]. For each class $c$, we compute $\omega_{c}=\max_{z=1,\ldots,R}s_{z,c}^{\text{local}}$ and $\psi_{c}=\min_{z=1,\ldots,R}s_{z,c}^{\text{local}}$, and define the aggregation score as $s_{c}^{\text{agg}}=\mathds{1}_{[\omega_{c}\geq\zeta^{\text{local}}]}\omega_{c}+\mathds{1}_{[\omega_{c}<\zeta^{\text{local}}]}\psi_{c}$; this yields the soft aggregation vector $S^{\text{agg}}=\{s_{1}^{\text{agg}},s_{2}^{\text{agg}},\ldots,s_{C}^{\text{agg}}\}$ for each input image.

Positive pseudo-labels.

To extract reliable positive pseudo-labels, we integrate both global and aggregated local similarities into $S^{final}=\frac{1}{2}\left(S^{global}+S^{agg}\right)$. Let $Q^{\prime}=\{l_{1}^{\prime},l_{2}^{\prime},\cdots,l_{C}^{\prime}\}$ be the pseudo labels of the image $I$. We convert the soft similarity scores $S^{final}$ into hard pseudo-labels as follows:

[TABLE]

where $\zeta^{global}$ is the global threshold for high-confidence positive pseudo-labels, set as a hyperparameter, and $k$ limits the number of positive pseudo-labels.

Negative pseudo-labels.

To identify potential negative pseudo-labels we compute average similarity scores as:

[TABLE]

We use $S^{\text{avg}}$ to refine $Q^{\prime}$ by assigning negative pseudo-labels, producing the final pseudo-labels $Q=\{l_{1},l_{2},\cdots,l_{C}\}$. A class $c$ is designated as a negative pseudo-label if its score $s_{c}^{\text{avg}}$ falls within the lowest $\Delta_{\text{neg}}\%$ of values in $S^{\text{avg}}$. Assuming a potential negative pseudo-label has low scores in both image $I$ and every patch $P_{z}$ according to the VLM, we define the assignment as:

[TABLE]

where $\theta_{\Delta_{\text{neg}}}(S^{\text{avg}})$ denotes the $\Delta_{\text{neg}}$-th-percentile of $S^{\text{avg}}$, serving as the threshold to identify the lowest $\Delta_{\text{neg}}\%$ of values in $S^{\text{avg}}$ as negative pseudo-labels.

6 Experiments

6.1 Setup

Datasets.

In this study, our method is evaluated through environmental experiments similar to those in [5, 13, 14, 2, 27] across four standard benchmark datasets: PASCAL VOC 2012 (VOC) [10], MS-COCO 2014 (COCO) [18], NUS-WIDE (NUS) [4], and CUB-200-2011 (CUB) [24]. From these fully labeled multi-label datasets, we simulate single positive training data by randomly retaining one positive label per training example, using the same seed as in [5]. Twenty percent of the training set for each dataset is set aside for validation purposes. Both the validation and test sets are fully labeled.

Implementation details.

For a fair comparison, we follow the standard SPML implementation described in [5, 14, 2, 27], using the ResNet-50 architecture [11], pre-trained on ImageNet [6]. Each image is resized to $448\times 448$ and augmented with random horizontal flipping. For the VLPL method in [27], we use CLIP ViT-L/16, while our approach uses CLIP ViT-B/16, both adopted from [22] to balance pseudo-labeling quality and training time. We train the models for 10 epochs on CUB and NUS, and 8 epochs on COCO and VOC, using the Adam optimizer [15]. The batch size is set to 8 for CUB and VOC, and 16 for NUS and COCO. A learning rate of $1\times 10^{-5}$ is used across all experiments. We evaluate performance using the mean Average Precision (mAP) metric, conducting the final evaluation on the test set with the model that achieves the highest performance on the withheld validation set, in line with previous methods [5, 13, 14, 2].

Baselines.

For a comprehensive understanding, we compare our proposed method, AEVLP, against several state-of-the-art methods within the SPML setting. Specifically, we evaluated against the following methods: Assume Negative [5], ROLE [5], EM [34], EM + APL [34], BoostLU + LL-R [14], SMILE [28], MINE [19], GR Loss [2], VLPL [27]. All methods above, including our AEVLP, were evaluated on the same benchmark datasets: VOC, COCO, NUS, and CUB for a fair comparison. The results were averaged over three runs, as recommended in previous studies [14].

6.2 Results and Discussion

Table 1 shows experimental results on four benchmark datasets: VOC, COCO, NUS, and CUB, using the previously described settings. Our method, AEVLP, achieves state-of-the-art performance across all datasets, demonstrating both effectiveness and robustness. Notably, it surpasses fully labeled multi-label classification on the VOC dataset. Although AEVLP can be seen as a fusion of GR Loss and VLPL, it not only combines their strengths but also surpasses both methods. VLPL, in particular, shows limitations in 3 out of 4 benchmark datasets compared with traditional state-of-the-art SPML methods. In contrast, AEVLP demonstrates superior performance, underscoring the effectiveness of introducing different focuses of the same input image to a VLM for pseudo-label generation. Notably, VLPL uses the CLIP model with a ViT Large backbone (428M parameters) [9], with $14\times 14$ patches and a 336-pixel resolution, whereas AEVLP achieves better results while using the CLIP model with a smaller ViT Base backbone (149.62M parameters) with $16\times 16$ patches and a $224\times 224$ input resolution (ViT-B/16). As can be seen from Table 1, traditional SPML methods [5, 28, 19, 2] based on pseudo-labeling approach struggle with datasets containing a large number of labels, such as CUB (312 labels). AEVLP, in contrast, shows a relative improvement of more than 15% over GR Loss method [2] and 3.62% over the next best-performing method, VLPL [27], extending GR Loss’s capability to work with pseudo-labels. Our proposed GPR Loss, described in Sec. 4.2), can cope with pseudo-labels generated by various mechanisms, as long as they provide an acceptable level of confidence.

6.3 Analysis

GPR Loss robustness.

GPR Loss is designed to work with various pseudo-labeling strategies, not only by effectively utilizing pseudo-labels but also by demonstrating robustness to noise. To validate this, we apply GPR Loss to pseudo-labels generated by different methods, including DAMP (AEVLP), Random Pseudo-labels, VLPL [27], and LL-Ct [13]. Random Pseudo-labels are obtained by randomly selecting unknown labels (assumed negative) as pseudo-positive labels, based on the average number per image in the dataset. VLPL provides pseudo-labels generated from the CLIP model, while LL-Ct converts assumed negative labels, particularly those with high loss values, into pseudo-positive labels. For the baseline of DAMP and Random Pseudo-labels, we use BCE for training; for the other pseudo-labeling methods, we replace their original loss function with GPR Loss. As shown in Tab. 2, incorporating GPR Loss consistently enhances mAP scores across all datasets for each pseudo-labeling strategy. This highlights GPR Loss’s effectiveness in mitigating the inherent noise in pseudo-labels, regardless of the generation method.

DAMP performance.

We apply pseudo-labeling to the training data of the VOC, COCO, NUS, and CUB datasets using the DAMP technique, evaluating over the same number of epochs used for training the main results in Tab. 1. Since pseudo-labels of a given image may vary across epochs, we report the results as average precision and recall across all epochs to assess the supervision quality that DAMP provides for training the backbone. For strict evaluation and to address label imbalance, we focus only on the missing positive labels in the training data according to the SPML setting, ignoring predictions for the confirmed single positive and true negative labels. To validate coverage of missing positive labels, we accumulate predictions across epochs: each new positive pseudo-label prediction is recorded, and precision and recall are computed on these accumulated positive labels. As illustrated in Tab. 3, DAMP, a strategy of randomization, enhances positive label coverage, with accumulated recall exceeding average recall, while maintaining a stable quality of pseudo-labels. Notably, precision across epochs varies only slightly from the average precision (see supplementary materials). In contrast, using fixed pseudo-labels results in lower coverage, and a straightforward accumulation approach introduces more noise.

Probability distribution analysis.

Figure 2 plots the distribution of the classifier output probabilities on the VOC test set from three different methods: Assume Negative, GR Loss, and AEVLP (ours). The ResNet-50 backbone is used in all three methods. Compared to existing method in Tab. 1, the Assume Negative method represents a naive approach, while GR Loss shows robustness across all datasets. As shown in Fig. 2(a), the Assume Negative method performs the worst: although it maintains high confidence for negative labels (close to 0), it includes numerous low-confidence predictions for positive labels, which can lead to significant errors. For both GR Loss in Fig. 2(b) and AEVLP in Fig. 2(c), the probability distributions of negative labels have a similar shape, concentrated around 0.2. However, AEVLP outperforms GR Loss, with its distribution of positive labels shifted closer to 1, even when learning from pseudo-labels that may introduce noise.

Mining negative pseudo-labels.

Fig. 3 shows the impact of varying the negative pseudo-label ratio $\Delta_{neg}\%$ on model performance. At approximately $20\%$ to $30\%$ negative pseudo-labels, the model achieves its best mAP performance across all datasets. This suggests that introducing a moderate proportion of negative pseudo-labels helps improve diversity in the learning process without significantly increasing noise. As the negative pseudo-label ratio increases further, the mAP slightly decreases but remains relatively stable. This is likely due to the influence of GPR Loss, which helps mitigate the negative impact of potentially inaccurate pseudo-labels in situations where there is an imbalance between positive and negative labels.

6.4 Ablation Study

Components of AEVLP.

In this section, we analyze the impact of each component in the AEVLP framework, as shown in Tab. 4. The ablation results in Tab. 4 shows that each component contributes to the overall performance improvement. Some observations are relevant here to describe the relationships between closely linked components that complement each other. The combination of noise addition $G(\cdot)$ with either augmentation $\operatorname{T}(\cdot)$ or overlapping $r$ provides a noticeable boost, as these pairs introduce valuable pseudo-labels that may be difficult to capture with the standard input alone. The positive pseudo-label re-weighting $v^{4}(p;\alpha)$ and regularization term $\mathcal{R}$ help stabilize learning from uncertain pseudo-labels. Additionally, the negative pseudo-label loss $\mathcal{L}^{3}_{n,i}$ encourages a stronger focus on potential negative labels, thereby enhancing discrimination.

Impact of the grid size.

We evaluate grid sizes $g$ (from 2 to 6) on four datasets: VOC (20 labels), COCO (80 labels), NUS (81 labels), and CUB (312 labels). The results (see Tab. 5) show that a grid size of 4 yields the best performance for VOC, COCO, and NUS, suggesting that a moderate grid size effectively balances spatial detail and computational efficiency. In contrast, CUB achieves its highest mAP with a grid size of 5, indicating that datasets with finer-grained labels benefit from slightly larger grid sizes to capture richer object details. While increasing grid size reduces the number of labels per patch, enhancing label separability, excessively small patches may fail to capture entire object instances and incur higher computational costs. These findings underscore the need to balance label reduction, object capture, and efficiency when selecting the optimal grid size.

Backbone architecture.

We evaluate several backbone architectures, as shown in Tab. 6. Specifically, we replace the baseline backbone with ConvNeXt-L [21], using two different pretrained versions (ImageNet-1K and ImageNet-22K), as well as ViT-L/14 [22]. The results demonstrate a notable improvement in performance with these architectures. This analysis confirms AEVLP’s flexibility, showing it can effectively incorporate advanced models to achieve substantial performance gains.

7 Conclusion

In this paper, we propose AEVLP, a novel framework for SPML. First, our approach introduces a new GPR loss function that effectively complements the pseudo-labels. We then present a dynamic augmented multi-focus pseudo-labeling strategy designed to overcome the limitations of fixed pseudo-labeling approaches, where the pseudo-labels for each image remain constant throughout training. Experimental results demonstrate that AEVLP achieves state-of-the-art performance on four benchmark datasets, reducing reliance on fully annotated data while addressing the shortcomings of traditional multi-label learning methods.