Seq-U-Net: A One-Dimensional Causal U-Net for Efficient Sequence Modelling

TL;DR

Seq-U-Net is an efficient causal U-Net architecture designed for sequence modeling, leveraging multi-scale features to improve long-term dependency modeling while reducing computational costs.

Contribution

The paper introduces Seq-U-Net, a novel causal U-Net architecture that enhances efficiency in sequence modeling by computing features at multiple time-scales.

Findings

Reduces memory and computation time compared to TCN and Wavenet.

Achieves over 4x speed-up in audio generation tasks.

Maintains comparable performance across various sequence modeling tasks.

Abstract

Convolutional neural networks (CNNs) with dilated filters such as the Wavenet or the Temporal Convolutional Network (TCN) have shown good results in a variety of sequence modelling tasks. However, efficiently modelling long-term dependencies in these sequences is still challenging. Although the receptive field of these models grows exponentially with the number of layers, computing the convolutions over very long sequences of features in each layer is time and memory-intensive, prohibiting the use of longer receptive fields in practice. To increase efficiency, we make use of the "slow feature" hypothesis stating that many features of interest are slowly varying over time. For this, we use a U-Net architecture that computes features at multiple time-scales and adapt it to our auto-regressive scenario by making convolutions causal. We apply our model ("Seq-U-Net") to a variety of tasks including language and audio generation. In comparison to TCN and Wavenet, our network consistently saves memory and computation time, with speed-ups for training and inference of over 4x in the audio generation experiment in particular, while achieving a comparable performance in all tasks.