Descriptor-Injected Cross-Modal Learning: A Systematic Exploration of Audio-MIDI Alignment via Spectral and Melodic Features

TL;DR

This paper systematically explores how hand-crafted spectral and melodic features can improve cross-modal audio-MIDI retrieval, demonstrating significant performance gains and insights into feature importance and model alignment.

Contribution

It introduces descriptor injection into cross-modal models, evaluates various configurations, and proposes reverse cross-attention to enhance audio-MIDI alignment.

Findings

Best configuration achieves 84.0% mean similarity, 8.8% above baseline.

Octave-band energy dynamics (A4) significantly influence model performance.

Descriptors increase transformer layer alignment, indicating better representational convergence.

Abstract

Cross-modal retrieval between audio recordings and symbolic music representations (MIDI) remains challenging because continuous waveforms and discrete event sequences encode different aspects of the same performance. We study descriptor injection, the augmentation of modality-specific encoders with hand-crafted domain features, as a bridge across this gap. In a three-phase campaign covering 13 descriptor-mechanism combinations, 6 architectural families, and 3 training schedules, the best configuration reaches a mean S of 84.0 percent across five independent seeds, improving the descriptor-free baseline by 8.8 percentage points. Causal ablation shows that the audio descriptor A4, based on octave-band energy dynamics, drives the gain in the top dual models, while the MIDI descriptor D4 has only a weak inference-time effect despite improving training dynamics. We also introduce reverse cross-attention, where descriptor tokens query encoder features, reducing attention operations relative to the standard formulation while remaining competitive. CKA analysis shows that descriptors substantially increase audio-MIDI transformer layer alignment, indicating representational convergence rather than simple feature concatenation. Perturbation analysis identifies high-frequency octave bands as the dominant discriminative signal. All experiments use MAESTRO v3.0.0 with an evaluation protocol controlling for composer and piece similarity.