# Audio Retrieval for Multimodal Design Documents: A New Dataset and Algorithms

## TL;DR

This paper introduces MELON, a large-scale dataset of multimodal design documents with paired audio, and proposes a novel cross-attention algorithm for retrieving relevant audio, improving multimodal content matching and accessibility.

## Contribution

The paper presents a new dataset, MELON, and a novel multimodal cross-attention retrieval algorithm for matching audio with design documents involving images and text.

## Key findings

- Our method outperforms existing state-of-the-art approaches.
- The dataset provides a new benchmark for multimodal audio retrieval.
- The approach enhances accessibility for visually impaired users.

## Abstract

We consider and propose a new problem of retrieving audio files relevant to multimodal design document inputs comprising both textual elements and visual imagery, e.g., birthday/greeting cards. In addition to enhancing user experience, integrating audio that matches the theme/style of these inputs also helps improve the accessibility of these documents (e.g., visually impaired people can listen to the audio instead). While recent work in audio retrieval exists, these methods and datasets are targeted explicitly towards natural images. However, our problem considers multimodal design documents (created by users using creative software) substantially different from a naturally clicked photograph. To this end, our first contribution is collecting and curating a new large-scale dataset called Melodic-Design (or MELON), comprising design documents representing various styles, themes, templates, illustrations, etc., paired with music audio. Given our paired image-text-audio dataset, our next contribution is a novel multimodal cross-attention audio retrieval (MMCAR) algorithm that enables training neural networks to learn a common shared feature space across image, text, and audio dimensions. We use these learned features to demonstrate that our method outperforms existing state-of-the-art methods and produce a new reference benchmark for the research community on our new dataset.