Performance Evaluation in Multimedia Retrieval
Loris Sauter, Ralph Gasser, Heiko Schuldt, Abraham Bernstein, Luca, Rossetto

TL;DR
This paper introduces a formal model and open-source infrastructure for multimedia retrieval evaluation to enhance experiment reproducibility and comparability across diverse use cases.
Contribution
It presents a comprehensive formal model and a flexible open-source tool for multimedia retrieval performance evaluation, addressing reproducibility challenges.
Findings
The model standardizes evaluation procedures.
The infrastructure supports diverse retrieval scenarios.
Improves reproducibility and comparability of experiments.
Abstract
Performance evaluation in multimedia retrieval, as in the information retrieval domain at large, relies heavily on retrieval experiments, employing a broad range of techniques and metrics. These can involve human-in-the-loop and machine-only settings for the retrieval process itself and the subsequent verification of results. Such experiments can be elaborate and use-case-specific, which can make them difficult to compare or replicate. In this paper, we present a formal model to express all relevant aspects of such retrieval experiments, as well as a flexible open-source evaluation infrastructure that implements the model. These contributions intend to make a step towards lowering the hurdles for conducting retrieval experiments and improving their reproducibility.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
