Graph Neural Networks as a Substitute for Transformers in Single-Cell Transcriptomics

TL;DR

This paper investigates replacing Transformers with Graph Neural Networks in single-cell transcriptomics, demonstrating GNNs can perform competitively while using fewer resources, especially when relative positional information is unnecessary.

Contribution

The study shows GNNs can substitute Transformers in position-agnostic tasks like single-cell transcriptomics, challenging the assumption that Transformers are always superior.

Findings

GNNs achieve comparable performance to Transformers on the dataset.

GNNs consume less computational resources than Transformers.

Position-agnostic data allows GNNs to be effective substitutes.

Abstract

Graph Neural Networks (GNNs) and Transformers share significant similarities in their encoding strategies for interacting with features from nodes of interest, where Transformers use query-key scores and GNNs use edges. Compared to GNNs, which are unable to encode relative positions, Transformers leverage dynamic attention capabilities to better represent relative relationships, thereby becoming the standard backbones in large-scale sequential pre-training. However, the subtle difference prompts us to consider: if positions are no longer crucial, could we substitute Transformers with Graph Neural Networks in some fields such as Single-Cell Transcriptomics? In this paper, we first explore the similarities and differences between GNNs and Transformers, specifically in terms of relative positions. Additionally, we design a synthetic example to illustrate their equivalence where there are no relative positions between tokens in the sample. Finally, we conduct extensive experiments on a large-scale position-agnostic dataset-single-cell transcriptomics-finding that GNNs achieve competitive performance compared to Transformers while consuming fewer computation resources. These findings provide novel insights for researchers in the field of single-cell transcriptomics, challenging the prevailing notion that the Transformer is always the optimum choice.