Performance Evaluation in Multimedia Retrieval

TL;DR

This paper introduces a formal model and open-source infrastructure for multimedia retrieval evaluation to enhance experiment reproducibility and comparability across diverse use cases.

Contribution

It presents a comprehensive formal model and a flexible open-source tool for multimedia retrieval performance evaluation, addressing reproducibility challenges.

Findings

The model standardizes evaluation procedures.

The infrastructure supports diverse retrieval scenarios.

Improves reproducibility and comparability of experiments.

Abstract

Performance evaluation in multimedia retrieval, as in the information retrieval domain at large, relies heavily on retrieval experiments, employing a broad range of techniques and metrics. These can involve human-in-the-loop and machine-only settings for the retrieval process itself and the subsequent verification of results. Such experiments can be elaborate and use-case-specific, which can make them difficult to compare or replicate. In this paper, we present a formal model to express all relevant aspects of such retrieval experiments, as well as a flexible open-source evaluation infrastructure that implements the model. These contributions intend to make a step towards lowering the hurdles for conducting retrieval experiments and improving their reproducibility.