ScDiVa: Masked Discrete Diffusion for Joint Modeling of Single-Cell Identity and Expression

TL;DR

scDiVa is a novel masked discrete diffusion model for single-cell RNA-seq data that improves joint modeling of gene identity and expression, overcoming autoregressive limitations and enhancing transfer learning performance.

Contribution

It introduces a masked discrete diffusion framework with a bidirectional denoiser and continuous-time masking, advancing joint modeling of gene identity and expression in single-cell analysis.

Findings

Pre-trained on 59 million cells.

Achieves strong transfer performance across benchmarks.

Outperforms autoregressive models in preserving global cell identity.

Abstract

Single-cell RNA-seq profiles are high-dimensional, sparse, and unordered, causing autoregressive generation to impose an artificial ordering bias and suffer from error accumulation. To address this, we propose scDiVa, a masked discrete diffusion foundation model that aligns generation with the dropout-like corruption process by defining a continuous-time forward masking mechanism in token space. ScDiVa features a bidirectional denoiser that jointly models discrete gene identities and continuous values, utilizing entropy-normalized serialization and a latent anchor token to maximize information efficiency and preserve global cell identity. The model is trained via depth-invariant time sampling and a dual denoising objective to simulate varying sparsity levels while ensuring precise recovery of both identity and magnitude. Pre-trained on 59 million cells, scDiVa achieves strong transfer performance across major benchmarks, including batch integration, cell type annotation, and perturbation response prediction. These results suggest that masked discrete diffusion serves as a biologically coherent and effective alternative to autoregression.