Transflower: probabilistic autoregressive dance generation with multimodal attention

TL;DR

This paper introduces Transflower, a probabilistic autoregressive model with multimodal attention for generating realistic dance movements conditioned on music, supported by a new large-scale 3D dance dataset.

Contribution

It presents a novel normalizing flow-based autoregressive architecture with multimodal transformer encoding for dance generation, and provides the largest 3D dance dataset for this task.

Findings

The model produces diverse, realistic dance movements matching music.

Multimodal attention improves dance-music alignment.

Probabilistic modeling enhances diversity in generated dances.

Abstract

Dance requires skillful composition of complex movements that follow rhythmic, tonal and timbral features of music. Formally, generating dance conditioned on a piece of music can be expressed as a problem of modelling a high-dimensional continuous motion signal, conditioned on an audio signal. In this work we make two contributions to tackle this problem. First, we present a novel probabilistic autoregressive architecture that models the distribution over future poses with a normalizing flow conditioned on previous poses as well as music context, using a multimodal transformer encoder. Second, we introduce the currently largest 3D dance-motion dataset, obtained with a variety of motion-capture technologies, and including both professional and casual dancers. Using this dataset, we compare our new model against two baselines, via objective metrics and a user study, and show that both the ability to model a probability distribution, as well as being able to attend over a large motion and music context are necessary to produce interesting, diverse, and realistic dance that matches the music.