Socializing the Semantic Gap: A Comparative Survey on Image Tag Assignment, Refinement and Retrieval

TL;DR

This survey reviews methods for image tag assignment, refinement, and retrieval, emphasizing social tagging and relevance estimation, and introduces a taxonomy and new experimental protocol for comparison.

Contribution

It presents a comprehensive taxonomy of image tagging methods, analyzes their information sources, and introduces a standardized experimental protocol for evaluating state-of-the-art approaches.

Findings

A new taxonomy clarifies different approaches and their merits.

Experimental results compare 11 methods across various datasets.

The survey highlights key challenges and future directions in social image tagging.

Abstract

Where previous reviews on content-based image retrieval emphasize on what can be seen in an image to bridge the semantic gap, this survey considers what people tag about an image. A comprehensive treatise of three closely linked problems, i.e., image tag assignment, refinement, and tag-based image retrieval is presented. While existing works vary in terms of their targeted tasks and methodology, they rely on the key functionality of tag relevance, i.e. estimating the relevance of a specific tag with respect to the visual content of a given image and its social context. By analyzing what information a specific method exploits to construct its tag relevance function and how such information is exploited, this paper introduces a taxonomy to structure the growing literature, understand the ingredients of the main works, clarify their connections and difference, and recognize their merits and limitations. For a head-to-head comparison between the state-of-the-art, a new experimental protocol is presented, with training sets containing 10k, 100k and 1m images and an evaluation on three test sets, contributed by various research groups. Eleven representative works are implemented and evaluated. Putting all this together, the survey aims to provide an overview of the past and foster progress for the near future.