Measuring and Predicting Tag Importance for Image Retrieval

TL;DR

This paper introduces a method to predict the importance of tags in image retrieval systems, leveraging visual, semantic, and contextual cues, leading to improved retrieval accuracy by aligning tag significance with visual content.

Contribution

It proposes a novel tag importance prediction model using SSVM and CCA, addressing the equal importance assumption in MIR systems to enhance retrieval performance.

Findings

Significant improvement in retrieval accuracy with the proposed method

Effective measurement of tag importance from image descriptions

Robustness of the model across three real-world datasets

Abstract

Textual data such as tags, sentence descriptions are combined with visual cues to reduce the semantic gap for image retrieval applications in today's Multimodal Image Retrieval (MIR) systems. However, all tags are treated as equally important in these systems, which may result in misalignment between visual and textual modalities during MIR training. This will further lead to degenerated retrieval performance at query time. To address this issue, we investigate the problem of tag importance prediction, where the goal is to automatically predict the tag importance and use it in image retrieval. To achieve this, we first propose a method to measure the relative importance of object and scene tags from image sentence descriptions. Using this as the ground truth, we present a tag importance prediction model to jointly exploit visual, semantic and context cues. The Structural Support Vector Machine (SSVM) formulation is adopted to ensure efficient training of the prediction model. Then, the Canonical Correlation Analysis (CCA) is employed to learn the relation between the image visual feature and tag importance to obtain robust retrieval performance. Experimental results on three real-world datasets show a significant performance improvement of the proposed MIR with Tag Importance Prediction (MIR/TIP) system over other MIR systems.