Fast and Low-Cost Genomic Foundation Models via Outlier Removal

TL;DR

GERM is a genomic foundation model that enhances efficiency and robustness by removing outliers, enabling faster adaptation, better quantization, and improved performance in resource-limited genomic modeling scenarios.

Contribution

The paper introduces GERM, a novel outlier removal technique for genomic models, improving adaptation speed, robustness, and efficiency over existing models like DNABERT-2.

Findings

GERM improves fine-tuning performance by 37.98%.

GERM enhances quantization robustness by 64.34%.

GERM reduces kurtosis by 92.14% and infinity norm by 82.77%.

Abstract

To address the challenge of scarce computational resources in genomic modeling, we introduce GERM, a genomic foundation model with strong compression performance and fast adaptability. GERM improves upon models like DNABERT-2 by eliminating outliers that hinder low-rank adaptation and post-training quantization, enhancing both efficiency and robustness. We replace the vanilla attention layer with an outlier-free mechanism inspired by associative memory models. By removing outliers during both pre-training and fine-tuning, this approach accelerates adaptation, reduces computational costs, and enhances quantization robustness within acceptable loss margins. Additionally, we propose GERM-T, a strategy that employs small-step continual learning within the outlier-free framework, leveraging original checkpoints to avoid retraining from scratch. Empirically, GERM improves fine-tuning performance by 37.98% and quantization by 64.34% over the baseline model. It also reduces average kurtosis by 92.14% and maximum infinity norm by 82.77%. Compared to leading methods, GERM consistently delivers superior performance, offering a practical solution for genomic modeling in resource-constrained settings. Code is available at https://github.com/MAGICS-LAB/GERM.