PanFoMa: A Lightweight Foundation Model and Benchmark for Pan-Cancer

TL;DR

PanFoMa introduces a lightweight hybrid neural network combining Transformers and state-space models for efficient, discriminative single-cell RNA sequencing analysis across pan-cancer datasets, with a comprehensive benchmark for evaluation.

Contribution

It presents PanFoMa, a novel hybrid model that balances performance and efficiency for pan-cancer single-cell analysis, along with a large-scale benchmark dataset.

Findings

Outperforms state-of-the-art models by 4.0% on the benchmark

Improves cell type annotation accuracy by 7.4%

Enhances batch and multi-omics integration results

Abstract

Single-cell RNA sequencing (scRNA-seq) is essential for decoding tumor heterogeneity. However, pan-cancer research still faces two key challenges: learning discriminative and efficient single-cell representations, and establishing a comprehensive evaluation benchmark. In this paper, we introduce PanFoMa, a lightweight hybrid neural network that combines the strengths of Transformers and state-space models to achieve a balance between performance and efficiency. PanFoMa consists of a front-end local-context encoder with shared self-attention layers to capture complex, order-independent gene interactions; and a back-end global sequential feature decoder that efficiently integrates global context using a linear-time state-space model. This modular design preserves the expressive power of Transformers while leveraging the scalability of Mamba to enable transcriptome modeling, effectively capturing both local and global regulatory signals. To enable robust evaluation, we also construct a large-scale pan-cancer single-cell benchmark, PanFoMaBench, containing over 3.5 million high-quality cells across 33 cancer subtypes, curated through a rigorous preprocessing pipeline. Experimental results show that PanFoMa outperforms state-of-the-art models on our pan-cancer benchmark (+4.0\%) and across multiple public tasks, including cell type annotation (+7.4\%), batch integration (+4.0\%) and multi-omics integration (+3.1\%). The code is available at https://github.com/Xiaoshui-Huang/PanFoMa.