An Overview of Cross-media Retrieval: Concepts, Methodologies, Benchmarks and Challenges

TL;DR

This paper provides a comprehensive overview of cross-media retrieval, discussing concepts, methodologies, benchmarks, and challenges, and introduces a new dataset to facilitate future research in this emerging field.

Contribution

It offers a detailed review of cross-media retrieval, introduces the first multi-media dataset XMedia, and provides benchmarks to support future algorithm development.

Findings

Reviewed over 100 references on cross-media retrieval

Constructed the first publicly available multi-media dataset XMedia

Provided benchmarks and experimental results for evaluation

Abstract

Multimedia retrieval plays an indispensable role in big data utilization. Past efforts mainly focused on single-media retrieval. However, the requirements of users are highly flexible, such as retrieving the relevant audio clips with one query of image. So challenges stemming from the "media gap", which means that representations of different media types are inconsistent, have attracted increasing attention. Cross-media retrieval is designed for the scenarios where the queries and retrieval results are of different media types. As a relatively new research topic, its concepts, methodologies and benchmarks are still not clear in the literatures. To address these issues, we review more than 100 references, give an overview including the concepts, methodologies, major challenges and open issues, as well as build up the benchmarks including datasets and experimental results. Researchers can directly adopt the benchmarks to promptly evaluate their proposed methods. This will help them to focus on algorithm design, rather than the time-consuming compared methods and results. It is noted that we have constructed a new dataset XMedia, which is the first publicly available dataset with up to five media types (text, image, video, audio and 3D model). We believe this overview will attract more researchers to focus on cross-media retrieval and be helpful to them.