Reparameterized Multi-Resolution Convolutions for Long Sequence Modelling

TL;DR

This paper introduces reparameterized multi-resolution convolutions ($\texttt{MRConv}$), a new method for efficiently learning expressive long-range kernels in sequence models, achieving state-of-the-art results across multiple data modalities.

Contribution

The paper proposes $\texttt{MRConv}$, a novel convolutional kernel parameterization that enhances long-range dependency learning through structural reparameterization and learnable kernel decay.

Findings

State-of-the-art results on Long Range Arena, Sequential CIFAR, and Speech Commands.

Improved ImageNet classification performance using $\texttt{MRConv}$.

Effective across various data modalities and models.

Abstract

Global convolutions have shown increasing promise as powerful general-purpose sequence models. However, training long convolutions is challenging, and kernel parameterizations must be able to learn long-range dependencies without overfitting. This work introduces reparameterized multi-resolution convolutions ($\texttt{MRConv}$), a novel approach to parameterizing global convolutional kernels for long-sequence modelling. By leveraging multi-resolution convolutions, incorporating structural reparameterization and introducing learnable kernel decay, $\texttt{MRConv}$ learns expressive long-range kernels that perform well across various data modalities. Our experiments demonstrate state-of-the-art performance on the Long Range Arena, Sequential CIFAR, and Speech Commands tasks among convolution models and linear-time transformers. Moreover, we report improved performance on ImageNet classification by replacing 2D convolutions with 1D $\texttt{MRConv}$ layers.