Kill Two Birds with One Stone: Weakly-Supervised Neural Network for Image Annotation and Tag Refinement

TL;DR

This paper introduces a weakly-supervised neural network that simultaneously learns image annotation and refines user-provided tags, effectively handling noisy labels in large-scale social image datasets.

Contribution

It presents a novel deep learning framework that integrates tag refinement with image annotation using weak supervision and batch-level constraints.

Findings

Achieves state-of-the-art performance on benchmark datasets.

Effectively handles noisy and incomplete user tags.

Demonstrates robustness and scalability for large-scale image sets.

Abstract

The number of social images has exploded by the wide adoption of social networks, and people like to share their comments about them. These comments can be a description of the image, or some objects, attributes, scenes in it, which are normally used as the user-provided tags. However, it is well-known that user-provided tags are incomplete and imprecise to some extent. Directly using them can damage the performance of related applications, such as the image annotation and retrieval. In this paper, we propose to learn an image annotation model and refine the user-provided tags simultaneously in a weakly-supervised manner. The deep neural network is utilized as the image feature learning and backbone annotation model, while visual consistency, semantic dependency, and user-error sparsity are introduced as the constraints at the batch level to alleviate the tag noise. Therefore, our model is highly flexible and stable to handle large-scale image sets. Experimental results on two benchmark datasets indicate that our proposed model achieves the best performance compared to the state-of-the-art methods.