Perceiving Longer Sequences With Bi-Directional Cross-Attention Transformers

TL;DR

The paper introduces BiXT, a bi-directional Transformer architecture that efficiently processes longer sequences across various modalities, outperforming larger models in speed and resource usage while maintaining competitive accuracy.

Contribution

BiXT replaces iterative attention with a bi-directional cross-attention module, enabling linear scaling with input size and simultaneous interpretation of semantics and location.

Findings

Outperforms larger models in vision tasks with 28% fewer FLOPs

Achieves up to 8.4x faster processing speed

Performs comparably to full Transformers on sequence tasks

Abstract

We present a novel bi-directional Transformer architecture (BiXT) which scales linearly with input size in terms of computational cost and memory consumption, but does not suffer the drop in performance or limitation to only one input modality seen with other efficient Transformer-based approaches. BiXT is inspired by the Perceiver architectures but replaces iterative attention with an efficient bi-directional cross-attention module in which input tokens and latent variables attend to each other simultaneously, leveraging a naturally emerging attention-symmetry between the two. This approach unlocks a key bottleneck experienced by Perceiver-like architectures and enables the processing and interpretation of both semantics ('what') and location ('where') to develop alongside each other over multiple layers -- allowing its direct application to dense and instance-based tasks alike. By combining efficiency with the generality and performance of a full Transformer architecture, BiXT can process longer sequences like point clouds, text or images at higher feature resolutions and achieves competitive performance across a range of tasks like point cloud part segmentation, semantic image segmentation, image classification, hierarchical sequence modeling and document retrieval. Our experiments demonstrate that BiXT models outperform larger competitors by leveraging longer sequences more efficiently on vision tasks like classification and segmentation, and perform on par with full Transformer variants on sequence modeling and document retrieval -- but require $28\%$ fewer FLOPs and are up to $8.4\times$ faster.