Context-Based Semantic-Aware Alignment for Semi-Supervised Multi-Label Learning

TL;DR

This paper introduces a context-based semantic-aware alignment approach for semi-supervised multi-label learning, leveraging vision-language models to improve label-specific feature extraction and pseudo-label quality.

Contribution

It proposes a novel framework that extracts label-specific features and uses a semi-supervised context identification task, enhancing semi-supervised multi-label learning with VLMs.

Findings

Significant improvement over existing methods on benchmark datasets.

Effective extraction of label-specific features enhances pseudo-label accuracy.

The approach leverages VLM knowledge to address limited labeled data challenges.

Abstract

Due to the lack of extensive precisely-annotated multi-label data in real word, semi-supervised multi-label learning (SSMLL) has gradually gained attention. Abundant knowledge embedded in vision-language models (VLMs) pre-trained on large-scale image-text pairs could alleviate the challenge of limited labeled data under SSMLL setting.Despite existing methods based on fine-tuning VLMs have achieved advances in weakly-supervised multi-label learning, they failed to fully leverage the information from labeled data to enhance the learning of unlabeled data. In this paper, we propose a context-based semantic-aware alignment method to solve the SSMLL problem by leveraging the knowledge of VLMs. To address the challenge of handling multiple semantics within an image, we introduce a novel framework design to extract label-specific image features. This design allows us to achieve a more compact alignment between text features and label-specific image features, leading the model to generate high-quality pseudo-labels. To incorporate the model with comprehensive understanding of image, we design a semi-supervised context identification auxiliary task to enhance the feature representation by capturing co-occurrence information. Extensive experiments on multiple benchmark datasets demonstrate the effectiveness of our proposed method.