WISE: A Multimodal Search Engine for Visual Scenes, Audio, Objects, Faces, Speech, and Metadata

TL;DR

WISE is an open-source multimodal search engine enabling users to perform complex, cross-modal queries on audiovisual data and metadata, supporting scalable retrieval and easy integration of new models.

Contribution

The paper introduces WISE, a versatile, scalable, and user-friendly multimodal search engine that integrates multiple retrieval modalities into a single open-source platform.

Findings

Supports natural-language and reverse-image queries across scenes and objects

Enables face, audio, speech, and metadata search with high scalability

Modular architecture allows easy integration of new models

Abstract

In this paper, we present WISE, an open-source audiovisual search engine which integrates a range of multimodal retrieval capabilities into a single, practical tool accessible to users without machine learning expertise. WISE supports natural-language and reverse-image queries at both the scene level (e.g. empty street) and object level (e.g. horse) across images and videos; face-based search for specific individuals; audio retrieval of acoustic events using text (e.g. wood creak) or an audio file; search over automatically transcribed speech; and filtering by user-provided metadata. Rich insights can be obtained by combining queries across modalities -- for example, retrieving German trains from a historical archive by applying the object query "train" and the metadata query "Germany", or searching for a face in a place. By employing vector search techniques, WISE can scale to support efficient retrieval over millions of images or thousands of hours of video. Its modular architecture facilitates the integration of new models. WISE can be deployed locally for private or sensitive collections, and has been applied to various real-world use cases. Our code is open-source and available at https://gitlab.com/vgg/wise/wise.