Understanding the Gist of Images - Ranking of Concepts for Multimedia Indexing

TL;DR

This paper presents a novel multimedia indexing approach that leverages external knowledge and a two-stage learning-to-rank framework to improve understanding of image content and enhance indexing accuracy.

Contribution

It introduces a two-stage learning-to-rank pipeline utilizing Wikipedia concepts and demonstrates its effectiveness on a large image dataset.

Findings

Achieved a MAP of 61.42 on MIRFlickr25k

Outperforms DBM in multimedia indexing

Competitively matches hashing-based methods

Abstract

Nowadays, where multimedia data is continuously generated, stored, and distributed, multimedia indexing, with its purpose of group- ing similar data, becomes more important than ever. Understanding the gist (=message) of multimedia instances is framed in related work as a ranking of concepts from a knowledge base, i.e., Wikipedia. We cast the task of multimedia indexing as a gist understanding problem. Our pipeline benefits from external knowledge and two subsequent learning- to-rank (l2r) settings. The first l2r produces a ranking of concepts rep- resenting the respective multimedia instance. The second l2r produces a mapping between the concept representation of an instance and the targeted class topic(s) for the multimedia indexing task. The evaluation on an established big size corpus (MIRFlickr25k, with 25,000 images), shows that multimedia indexing benefits from understanding the gist. Finally, with a MAP of 61.42, it can be shown that the multimedia in- dexing task benefits from understanding the gist. Thus, the presented end-to-end setting outperforms DBM and competes with Hashing-based methods.