Systematic Evaluation of Single-Cell Foundation Model Interpretability Reveals Attention Captures Co-Expression Rather Than Unique Regulatory Signal

TL;DR

This paper systematically evaluates single-cell foundation model interpretability, revealing that attention captures co-expression rather than unique regulatory signals, and introduces CSSI to improve gene regulatory network recovery.

Contribution

It provides a comprehensive evaluation framework and uncovers that attention patterns do not enhance perturbation prediction, challenging assumptions about interpretability in single-cell models.

Findings

Attention encodes biological information with layer-specific organization.

Attention and correlation edges do not improve perturbation prediction.

CSSI improves gene regulatory network recovery by up to 1.85x.

Abstract

We present a systematic evaluation framework - thirty-seven analyses, 153 statistical tests, four cell types, two perturbation modalities - for assessing mechanistic interpretability in single-cell foundation models. Applying this framework to scGPT and Geneformer, we find that attention patterns encode structured biological information with layer-specific organisation - protein-protein interactions in early layers, transcriptional regulation in late layers - but this structure provides no incremental value for perturbation prediction: trivial gene-level baselines outperform both attention and correlation edges (AUROC 0.81-0.88 versus 0.70), pairwise edge scores add zero predictive contribution, and causal ablation of regulatory heads produces no degradation. These findings generalise from K562 to RPE1 cells; the attention-correlation relationship is context-dependent, but gene-level dominance is universal. Cell-State Stratified Interpretability (CSSI) addresses an attention-specific scaling failure, improving GRN recovery up to 1.85x. The framework establishes reusable quality-control standards for the field.