BumbleBee: A Transformer for Music

TL;DR

BumbleBee is a novel transformer model designed for MIDI music generation, utilizing dilating sliding window attention to handle long sequences, and benchmarked against existing models like Music Transformer and LSTM.

Contribution

Introduces BumbleBee, a transformer with dilating sliding window attention for improved long-sequence music generation, and provides comparative analysis with existing models.

Findings

BumbleBee outperforms LSTM and Music Transformer on MIDI data.

Dilating sliding window attention improves long-sequence modeling.

Benchmark results demonstrate competitive performance.

Abstract

We will introduce BumbleBee, a transformer model that will generate MIDI music data . We will tackle the issue of transformers applied to long sequences by implementing a longformer generative model that uses dilating sliding windows to compute the attention layers. We will compare our results to that of the music transformer and Long-Short term memory (LSTM) to benchmark our results. This analysis will be performed using piano MIDI files, in particular , the JSB Chorales dataset that has already been used for other research works (Huang et al., 2018)