SymphonyGen: 3D Hierarchical Orchestral Generation with Controllable Harmony Skeleton

TL;DR

SymphonyGen is a hierarchical model for orchestral music generation that combines structural control, harmonic refinement, and perceptual alignment to produce more musical and preferred symphonic outputs.

Contribution

It introduces a novel hierarchical framework with a beat-quantized harmony skeleton and reinforcement learning techniques for improved orchestral music synthesis.

Findings

Objective evaluations show improved harmonic cleanliness.

Subjective evaluations favor SymphonyGen over baselines.

Dissonance-averse sampling reduces tonal clashes.

Abstract

Generating symphonic music requires simultaneously managing high-level structural form and dense, multi-track orchestration. Existing symbolic models often struggle with a "complexity-control imbalance", in which scaling bottlenecks limit long-term granular steerability. We present SymphonyGen, a 3D hierarchical framework for contemporary cinematic orchestration. SymphonyGen employs a cascading decoder architecture that decomposes the Bar, Track, and Event axes, improving computational efficiency and scalability over conventional 1D or 2D models. We introduce "short-score" conditioning via a beat-quantized multi-voice harmony skeleton, enabling outline control while preserving textural diversity. The model is further refined using Group Relative Policy Optimization (GRPO) with a cross-modal audio-perceptual reward, aligning symbolic output with modern acoustic expectations. Additionally, we implement a dissonance-averse sampling algorithm to suppress unintended tonal clashes during inference. Objective evaluations show that both reinforcement learning and dissonance-averse sampling effectively enhance harmonic cleanliness while maintaining melodic expression. Subjective evaluations demonstrate that SymphonyGen outperforms baselines in musicality and preference for orchestral music generation. Demo page: https://symphonygen.github.io/