Do Music Generation Models Encode Music Theory?

TL;DR

This paper investigates whether music foundation models encode fundamental music theory concepts by introducing SynTheory, a synthetic dataset, and probing models like Jukebox and MusicGen, revealing that these concepts are detectable within their internal representations.

Contribution

The paper introduces SynTheory, a comprehensive synthetic dataset for music theory concepts, and presents a framework to probe and analyze how music foundation models encode these concepts internally.

Findings

Music theory concepts are detectable within foundation models.

Model size and layer influence the degree of encoding.

Encoding strength varies across different music theory elements.

Abstract

Music foundation models possess impressive music generation capabilities. When people compose music, they may infuse their understanding of music into their work, by using notes and intervals to craft melodies, chords to build progressions, and tempo to create a rhythmic feel. To what extent is this true of music generation models? More specifically, are fundamental Western music theory concepts observable within the "inner workings" of these models? Recent work proposed leveraging latent audio representations from music generation models towards music information retrieval tasks (e.g. genre classification, emotion recognition), which suggests that high-level musical characteristics are encoded within these models. However, probing individual music theory concepts (e.g. tempo, pitch class, chord quality) remains under-explored. Thus, we introduce SynTheory, a synthetic MIDI and audio music theory dataset, consisting of tempos, time signatures, notes, intervals, scales, chords, and chord progressions concepts. We then propose a framework to probe for these music theory concepts in music foundation models (Jukebox and MusicGen) and assess how strongly they encode these concepts within their internal representations. Our findings suggest that music theory concepts are discernible within foundation models and that the degree to which they are detectable varies by model size and layer.