A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomics Representations through Morphological Features

TL;DR

This paper introduces a novel framework that enhances transcriptomics data by distilling knowledge from microscopy images using weakly paired datasets, incorporating semi-clipped and PEA augmentation techniques to improve biological representations.

Contribution

It presents a new cross-modal distillation method with semi-clipped and PEA augmentation, advancing transcriptomics analysis with morphological features.

Findings

Achieved state-of-the-art results with semi-clipped CLIP adaptation.

Enhanced transcriptomics data quality using PEA augmentation.

Improved biological task performance with richer unimodal representations.

Abstract

Understanding cellular responses to stimuli is crucial for biological discovery and drug development. Transcriptomics provides interpretable, gene-level insights, while microscopy imaging offers rich predictive features but is harder to interpret. Weakly paired datasets, where samples share biological states, enable multimodal learning but are scarce, limiting their utility for training and multimodal inference. We propose a framework to enhance transcriptomics by distilling knowledge from microscopy images. Using weakly paired data, our method aligns and binds modalities, enriching gene expression representations with morphological information. To address data scarcity, we introduce (1) Semi-Clipped, an adaptation of CLIP for cross-modal distillation using pretrained foundation models, achieving state-of-the-art results, and (2) PEA (Perturbation Embedding Augmentation), a novel augmentation technique that enhances transcriptomics data while preserving inherent biological information. These strategies improve the predictive power and retain the interpretability of transcriptomics, enabling rich unimodal representations for complex biological tasks.