Harnessing the Power of Single-Cell Large Language Models with Parameter Efficient Fine-Tuning using scPEFT

TL;DR

This paper introduces scPEFT, a method to efficiently fine-tune single-cell large language models for better performance in specific biological tasks with limited data.

Contribution

scPEFT introduces low-dimensional adapters for parameter-efficient fine-tuning of scLLMs, reducing memory and parameter usage significantly.

Findings

scPEFT outperformed zero-shot and traditional fine-tuning in disease-specific and cross-species tasks.

It reduced GPU memory usage by over half and parameter tuning by more than 96%.

Attention analysis revealed COVID-related genes and unique blood cell subpopulations.

Abstract

Single-cell large language models (scLLMs) capture essential biological insights from vast single-cell atlases but struggle in out-of-context applications, where zero-shot predictions can be unreliable. To address this, we introduce a single-cell parameter-efficient fine-tuning (scPEFT) framework that integrates learnable, low-dimensional adapters into scLLMs. By freezing the backbone model and updating only the adapter parameters, scPEFT efficiently adapts to specific tasks using limited custom data. This approach mitigates catastrophic forgetting, reduces parameter tuning by over 96%, and decreases GPU memory usage by more than half, significantly enhancing scLLMs’s accessibility for resource-constrained researchers. Validated across diverse datasets, scPEFT outperformed zero-shot models and traditional fine-tuning in disease-specific, cross-species, and under-characterized cell population tasks. Its attention-mechanism analysis identified COVID-related genes associated with specific cell states and uncovered unique blood cell subpopulations, demonstrating scPEFT’s capacity for condition-specific interpretations. These findings position scPEFT as an efficient solution for improving scLLMs’ utilities in general single-cell analyses.

Figures