Emotion-Conditioned Melody Harmonization with Hierarchical Variational Autoencoder

TL;DR

This paper introduces a hierarchical variational autoencoder model that conditions melody harmonization on emotions, enhancing harmony quality and variability by modeling global and local musical properties with attention mechanisms.

Contribution

The novel LHVAE model incorporates emotional conditions at multiple levels and uses attention to improve emotion-aware melody harmonization and variability.

Findings

Outperforms other LSTM-based models in objective metrics

Subjective evaluation shows emotion remains consistent despite chord changes

Qualitative analysis confirms the model's ability to generate diverse harmonies

Abstract

Existing melody harmonization models have made great progress in improving the quality of generated harmonies, but most of them ignored the emotions beneath the music. Meanwhile, the variability of harmonies generated by previous methods is insufficient. To solve these problems, we propose a novel LSTM-based Hierarchical Variational Auto-Encoder (LHVAE) to investigate the influence of emotional conditions on melody harmonization, while improving the quality of generated harmonies and capturing the abundant variability of chord progressions. Specifically, LHVAE incorporates latent variables and emotional conditions at different levels (piece- and bar-level) to model the global and local music properties. Additionally, we introduce an attention-based melody context vector at each step to better learn the correspondence between melodies and harmonies. Objective experimental results show that our proposed model outperforms other LSTM-based models. Through subjective evaluation, we conclude that only altering the types of chords hardly changes the overall emotion of the music. The qualitative analysis demonstrates the ability of our model to generate variable harmonies.