MoRE: Batch-Robust Multi-Omics Representations from Frozen Pre-trained Transformers

TL;DR

MoRE introduces a parameter-efficient framework using frozen pre-trained transformers with adapters for robust multi-omics data integration, generalizing across cell types and platforms with minimal fine-tuning.

Contribution

MoRE repurposes frozen transformers with lightweight adapters and novel loss functions for effective multi-omics integration, reducing training complexity and enhancing cross-modality generalization.

Findings

Achieves competitive batch robustness and biological conservation.

Reduces trainable parameters compared to fully fine-tuned models.

Outperforms baseline methods in integration and rare population detection.

Abstract

Representation learning on multi-omics data is challenging due to extreme dimensionality, modality heterogeneity, and cohort-specific batch effects. While pre-trained transformer backbones have shown broad generalization capabilities in biological sequence modeling, their application to multi-omics integration remains underexplored. We present MoRE (Multi-Omics Representation Embedding), a framework that repurposes frozen pre-trained transformers to align heterogeneous assays into a shared latent space. Unlike purely generative approaches, MoRE employs a parameter-efficient fine-tuning (PEFT) strategy, prioritizing cross-sample and cross-modality alignment over simple sequence reconstruction. Specifically, MoRE attaches lightweight, modality-specific adapters and a task-adaptive fusion layer to the frozen backbone. It optimizes a masked modeling objective jointly with supervised contrastive and batch-invariant alignment losses, yielding structure-preserving embeddings that generalize across unseen cell types and platforms. We benchmark MoRE against established baselines, including scGPT, scVI, and Harmony with Scrublet, evaluating integration fidelity, rare population detection, and modality transfer. Our results demonstrate that MoRE achieves competitive batch robustness and biological conservation while significantly reducing trainable parameters compared to fully fine-tuned models. This work positions MoRE as a practical step toward general-purpose omics foundation models.