LLM2Fx-Tools: Tool Calling For Music Post-Production

TL;DR

This paper presents LLM2Fx-Tools, a novel framework that leverages large language models to generate and control audio effect chains for music post-production, enabling interpretable and style transfer capabilities.

Contribution

It introduces a new multimodal tool-calling framework and a structured dataset for audio effects, advancing LLM applications in music production.

Findings

Accurately infers audio effect chains from processed and unprocessed audio.

Enables style transfer of audio effects from reference to new content.

Demonstrates effective reasoning and control in music post-production tasks.

Abstract

This paper introduces LLM2Fx-Tools, a multimodal tool-calling framework that generates executable sequences of audio effects (Fx-chain) for music post-production. LLM2Fx-Tools uses a large language model (LLM) to understand audio inputs, select audio effects types, determine their order, and estimate parameters, guided by chain-of-thought (CoT) planning. We also present LP-Fx, a new instruction-following dataset with structured CoT annotations and tool calls for audio effects modules. Experiments show that LLM2Fx-Tools can infer an Fx-chain and its parameters from pairs of unprocessed and processed audio, enabled by autoregressive sequence modeling, tool calling, and CoT reasoning. We further validate the system in a style transfer setting, where audio effects information is transferred from a reference source and applied to new content. Finally, LLM-as-a-judge evaluation demonstrates that our approach generates appropriate CoT reasoning and responses for music production queries. To our knowledge, this is the first work to apply LLM-based tool calling to audio effects modules, enabling interpretable and controllable music production.