Epigenomic language models powered by Cerebras

TL;DR

This paper introduces EBERT, a novel Transformer-based model that integrates DNA sequences and epigenetic data to improve cell type-specific gene regulation predictions, enabled by large-scale pre-training on the human genome with Cerebras hardware.

Contribution

The paper presents the first large-scale pre-trained epigenomic language model that combines DNA and epigenetic data, demonstrating enhanced transfer learning for biological tasks.

Findings

EBERT outperforms previous models on multiple benchmarks.

Inclusion of epigenetic data improves prediction accuracy.

Cerebras hardware enabled training of a complex model at scale.

Abstract

Large scale self-supervised pre-training of Transformer language models has advanced the field of Natural Language Processing and shown promise in cross-application to the biological `languages' of proteins and DNA. Learning effective representations of DNA sequences using large genomic sequence corpuses may accelerate the development of models of gene regulation and function through transfer learning. However, to accurately model cell type-specific gene regulation and function, it is necessary to consider not only the information contained in DNA nucleotide sequences, which is mostly invariant between cell types, but also how the local chemical and structural `epigenetic state' of chromosomes varies between cell types. Here, we introduce a Bidirectional Encoder Representations from Transformers (BERT) model that learns representations based on both DNA sequence and paired epigenetic state inputs, which we call Epigenomic BERT (or EBERT). We pre-train EBERT with a masked language model objective across the entire human genome and across 127 cell types. Training this complex model with a previously prohibitively large dataset was made possible for the first time by a partnership with Cerebras Systems, whose CS-1 system powered all pre-training experiments. We show EBERT's transfer learning potential by demonstrating strong performance on a cell type-specific transcription factor binding prediction task. Our fine-tuned model exceeds state of the art performance on 4 of 13 evaluation datasets from ENCODE-DREAM benchmarks and earns an overall rank of 3rd on the challenge leaderboard. We explore how the inclusion of epigenetic data and task specific feature augmentation impact transfer learning performance.