Converting Transformers into DGNNs Form

TL;DR

This paper introduces Converter, a novel method that transforms Transformers into DGNNs using digraph convolution, achieving superior performance and efficiency on various benchmarks.

Contribution

It formalizes a synthetic digraph convolution to convert Transformers into DGNNs, offering a lightweight and effective alternative to self-attention.

Findings

Converter outperforms traditional Transformers on benchmarks.

It maintains computational efficiency and simplicity.

Effective across diverse tasks like document and DNA classification.

Abstract

Recent advances in deep learning have established Transformer architectures as the predominant modeling paradigm. Central to the success of Transformers is the self-attention mechanism, which scores the similarity between query and key matrices to modulate a value matrix. This operation bears striking similarities to digraph convolution, prompting an investigation into whether digraph convolution could serve as an alternative to self-attention. In this study, we formalize this concept by introducing a synthetic unitary digraph convolution based on the digraph Fourier transform. The resulting model, which we term Converter, effectively converts a Transformer into a Directed Graph Neural Network (DGNN) form. We have tested Converter on Long-Range Arena benchmark, long document classification, and DNA sequence-based taxonomy classification. Our experimental results demonstrate that Converter achieves superior performance while maintaining computational efficiency and architectural simplicity, which establishes it as a lightweight yet powerful Transformer variant.