Finding the Topic of a Set of Images

TL;DR

This paper presents a novel method for identifying the overarching topic of a set of images by leveraging visual similarity, noisy crowd-sourced tags, and a word refinement process, outperforming baseline approaches.

Contribution

The paper introduces a new approach combining image retrieval, word selection via random walks, and a CRF-based word mapping to determine image set topics, handling noisy tags and diverse images.

Findings

The proposed algorithm outperforms baseline methods across 300 topics.

The CRF word mapping improves semantic consistency of tags.

The method effectively handles noisy, diverse image tags from large datasets.

Abstract

In this paper we introduce the problem of determining the topic that a set of images is describing, where every topic is represented as a set of words. Different from other problems like tag assignment or similar, a) we assume multiple images are used as input instead of single image, b) Input images are typically not visually related, c) Input images are not necessarily semantically close, and d) Output word space is unconstrained. In our proposed solution, visual information of each query image is used to retrieve similar images with text labels (tags) from an image database. We consider a scenario where the tags are very noisy and diverse, given that they were obtained by implicit crowd-sourcing in a database of 1 million images and over seventy seven thousand tags. The words or tags associated to each query are processed jointly in a word selection algorithm using random walks that allows to refine the search topic, rejecting words that are not part of the topic and produce a set of words that fairly describe the topic. Experiments on a dataset of 300 topics, with up to twenty images per topic, show that our algorithm performs better than the proposed baseline for any number of query images. We also present a new Conditional Random Field (CRF) word mapping algorithm that preserves the semantic similarity of the mapped words, increasing the performance of the results over the baseline.