Device-Guided Music Transfer

TL;DR

DeMT is a novel method that uses a vision-language model to extract device-specific embeddings from speaker frequency responses, enabling effective device-style transfer and adaptation in music playback.

Contribution

It introduces a new approach combining vision-language models and transformers to adapt music to unseen speaker devices based on their frequency response curves.

Findings

Enables effective speaker-style transfer.

Supports robust few-shot adaptation for unseen devices.

Improves quality and style augmentation in music playback.

Abstract

Device-guided music transfer adapts playback across unseen devices for users who lack them. Existing methods mainly focus on modifying the timbre, rhythm, harmony, or instrumentation to mimic genres or artists, overlooking the diverse hardware properties of the playback device (i.e., speaker). Therefore, we propose DeMT, which processes a speaker's frequency response curve as a line graph using a vision-language model to extract device embeddings. These embeddings then condition a hybrid transformer via feature-wise linear modulation. Fine-tuned on a self-collected dataset, DeMT enables effective speaker-style transfer and robust few-shot adaptation for unseen devices, supporting applications like device-style augmentation and quality enhancement.