DeepResonance: Enhancing Multimodal Music Understanding via Music-centric Multi-way Instruction Tuning

TL;DR

DeepResonance is a multimodal music understanding model that integrates music, text, images, and videos through multi-way instruction tuning, achieving state-of-the-art results across various tasks.

Contribution

The paper introduces DeepResonance, a novel multimodal music understanding LLM with multi-way instruction tuning and new datasets, enhancing integration of visual and textual music features.

Findings

Achieves state-of-the-art performance on six music understanding tasks.

Effectively integrates visual and textual modalities for improved understanding.

Demonstrates the benefits of auxiliary modalities in music comprehension.

Abstract

Recent advancements in music large language models (LLMs) have significantly improved music understanding tasks, which involve the model's ability to analyze and interpret various musical elements. These improvements primarily focused on integrating both music and text inputs. However, the potential of incorporating additional modalities such as images, videos and textual music features to enhance music understanding remains unexplored. To bridge this gap, we propose DeepResonance, a multimodal music understanding LLM fine-tuned via multi-way instruction tuning with multi-way aligned music, text, image, and video data. To this end, we construct Music4way-MI2T, Music4way-MV2T, and Music4way-Any2T, three 4-way training and evaluation datasets designed to enable DeepResonance to integrate both visual and textual music feature content. We also introduce multi-sampled ImageBind embeddings and a pre-LLM fusion Transformer to enhance modality fusion prior to input into text LLMs, tailoring for multi-way instruction tuning. Our model achieves state-of-the-art performances across six music understanding tasks, highlighting the benefits of the auxiliary modalities and the structural superiority of DeepResonance. We open-source the codes, models and datasets we constructed: github.com/sony/DeepResonance.