Multi-Dimensional Hyena for Spatial Inductive Bias

TL;DR

This paper introduces a data-efficient vision transformer using a novel Hyena N-D layer that enhances performance without relying on self-attention, especially effective on small datasets and when combined with traditional attention layers.

Contribution

The paper proposes a new Hyena N-D layer for vision transformers, improving data efficiency and performance without self-attention, and explores hybrid models combining Hyena and attention mechanisms.

Findings

Hyena N-D layer improves various Vision Transformer architectures.

Hyena-based ViT outperforms recent small dataset models.

Hybrid Hyena-attention models boost overall performance.

Abstract

In recent years, Vision Transformers have attracted increasing interest from computer vision researchers. However, the advantage of these transformers over CNNs is only fully manifested when trained over a large dataset, mainly due to the reduced inductive bias towards spatial locality within the transformer's self-attention mechanism. In this work, we present a data-efficient vision transformer that does not rely on self-attention. Instead, it employs a novel generalization to multiple axes of the very recent Hyena layer. We propose several alternative approaches for obtaining this generalization and delve into their unique distinctions and considerations from both empirical and theoretical perspectives. Our empirical findings indicate that the proposed Hyena N-D layer boosts the performance of various Vision Transformer architectures, such as ViT, Swin, and DeiT across multiple datasets. Furthermore, in the small dataset regime, our Hyena-based ViT is favorable to ViT variants from the recent literature that are specifically designed for solving the same challenge, i.e., working with small datasets or incorporating image-specific inductive bias into the self-attention mechanism. Finally, we show that a hybrid approach that is based on Hyena N-D for the first layers in ViT, followed by layers that incorporate conventional attention, consistently boosts the performance of various vision transformer architectures.