Extending Cross-Modal Retrieval with Interactive Learning to Improve Image Retrieval Performance in Forensics

TL;DR

This paper introduces Excalibur, a zero-shot cross-modal image retrieval system enhanced with interactive learning, significantly improving forensic image retrieval performance and user experience in handling large unstructured digital evidence.

Contribution

The paper presents Excalibur, a novel interactive learning extension to zero-shot cross-modal retrieval, tailored for forensic image analysis, demonstrating improved accuracy and usability.

Findings

Interactive learning significantly enhances retrieval performance.

Users find Excalibur effective and easy to use.

Participants are interested in adopting Excalibur in practice.

Abstract

Nowadays, one of the critical challenges in forensics is analyzing the enormous amounts of unstructured digital evidence, such as images. Often, unstructured digital evidence contains precious information for forensic investigations. Therefore, a retrieval system that can effectively identify forensically relevant images is paramount. In this work, we explored the effectiveness of interactive learning in improving image retrieval performance in the forensic domain by proposing Excalibur - a zero-shot cross-modal image retrieval system extended with interactive learning. Excalibur was evaluated using both simulations and a user study. The simulations reveal that interactive learning is highly effective in improving retrieval performance in the forensic domain. Furthermore, user study participants could effectively leverage the power of interactive learning. Finally, they considered Excalibur effective and straightforward to use and expressed interest in using it in their daily practice.