GeneMask: Fast Pretraining of Gene Sequences to Enable Few-Shot Learning

TL;DR

GeneMask introduces a novel masking algorithm for gene sequence pretraining that leverages local span selection based on NPMI, significantly improving few-shot classification performance with less training time.

Contribution

The paper proposes GeneMask, a new masking method for gene sequence MLM training that outperforms existing models in few-shot tasks and captures latent genomic information.

Findings

GeneMask outperforms DNABert and LOGO on four gene classification datasets.

GeneMask requires less than one-tenth of the training epochs of previous models.

High PMI tokens correlate with conserved DNA motifs.

Abstract

Large-scale language models such as DNABert and LOGO aim to learn optimal gene representations and are trained on the entire Human Reference Genome. However, standard tokenization schemes involve a simple sliding window of tokens like k-mers that do not leverage any gene-based semantics and thus may lead to (trivial) masking of easily predictable sequences and subsequently inefficient Masked Language Modeling (MLM) training. Therefore, we propose a novel masking algorithm, GeneMask, for MLM training of gene sequences, where we randomly identify positions in a gene sequence as mask centers and locally select the span around the mask center with the highest Normalized Pointwise Mutual Information (NPMI) to mask. We observe that in the absence of human-understandable semantics in the genomics domain (in contrast, semantic units like words and phrases are inherently available in NLP), GeneMask-based models substantially outperform the SOTA models (DNABert and LOGO) over four benchmark gene sequence classification datasets in five few-shot settings (10 to 1000-shot). More significantly, the GeneMask-based DNABert model is trained for less than one-tenth of the number of epochs of the original SOTA model. We also observe a strong correlation between top-ranked PMI tokens and conserved DNA sequence motifs, which may indicate the incorporation of latent genomic information. The codes (including trained models) and datasets are made publicly available at https://github.com/roysoumya/GeneMask.