Attending to Topological Spaces: The Cellular Transformer
Rub\'en Ballester, Pablo Hern\'andez-Garc\'ia, Mathilde Papillon,, Claudio Battiloro, Nina Miolane, Tolga Birdal, Carles Casacuberta, Sergio, Escalera, Mustafa Hajij

TL;DR
The paper introduces the Cellular Transformer, a novel neural network architecture that leverages topological structures like cell complexes to improve predictive performance on graph datasets, achieving state-of-the-art results.
Contribution
It presents a new transformer architecture tailored for cell complexes, with specialized attention mechanisms and positional encodings, advancing topological deep learning methods.
Findings
Achieves state-of-the-art performance on graph datasets.
Operates effectively without complex enhancements.
Leverages topological structures for improved learning.
Abstract
Topological Deep Learning seeks to enhance the predictive performance of neural network models by harnessing topological structures in input data. Topological neural networks operate on spaces such as cell complexes and hypergraphs, that can be seen as generalizations of graphs. In this work, we introduce the Cellular Transformer (CT), a novel architecture that generalizes graph-based transformers to cell complexes. First, we propose a new formulation of the usual self- and cross-attention mechanisms, tailored to leverage incidence relations in cell complexes, e.g., edge-face and node-edge relations. Additionally, we propose a set of topological positional encodings specifically designed for cell complexes. By transforming three graph datasets into cell complex datasets, our experiments reveal that CT not only achieves state-of-the-art performance, but it does so without the need for…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsCellular Automata and Applications
MethodsAttention Is All You Need · Sparse Evolutionary Training · Linear Layer · Position-Wise Feed-Forward Layer · Multi-Head Attention · Residual Connection · Byte Pair Encoding · Label Smoothing · Adam · Absolute Position Encodings
