# Single-Cell Multimodal Prediction via Transformers

## TL;DR

This paper introduces scMoFormer, a transformer-based framework for multimodal single-cell data analysis that leverages downstream task information and external knowledge, outperforming existing methods in benchmarks and winning a Kaggle competition.

## Contribution

The work presents a novel transformer-based model for multimodal single-cell data that incorporates downstream task info and external knowledge, addressing limitations of previous GNN-based methods.

## Key findings

- scMoFormer outperforms existing models on benchmark datasets.
- Achieved top 2% ranking in a NeurIPS 2022 Kaggle competition.
- Successfully incorporates external domain knowledge into multimodal analysis.

## Abstract

The recent development of multimodal single-cell technology has made the possibility of acquiring multiple omics data from individual cells, thereby enabling a deeper understanding of cellular states and dynamics. Nevertheless, the proliferation of multimodal single-cell data also introduces tremendous challenges in modeling the complex interactions among different modalities. The recently advanced methods focus on constructing static interaction graphs and applying graph neural networks (GNNs) to learn from multimodal data. However, such static graphs can be suboptimal as they do not take advantage of the downstream task information; meanwhile GNNs also have some inherent limitations when deeply stacking GNN layers. To tackle these issues, in this work, we investigate how to leverage transformers for multimodal single-cell data in an end-to-end manner while exploiting downstream task information. In particular, we propose a scMoFormer framework which can readily incorporate external domain knowledge and model the interactions within each modality and cross modalities. Extensive experiments demonstrate that scMoFormer achieves superior performance on various benchmark datasets. Remarkably, scMoFormer won a Kaggle silver medal with the rank of 24/1221 (Top 2%) without ensemble in a NeurIPS 2022 competition. Our implementation is publicly available at Github.