Cross-Granularity Representations for Biological Sequences: Insights from ESM and BiGCARP

TL;DR

This paper explores how integrating hierarchical biological sequence representations from models like ESM and BiGCARP can enhance understanding and performance in biological tasks by leveraging complementary information across different levels of biological organization.

Contribution

It demonstrates the benefits of combining cross-granularity embeddings from different models, revealing their complementary nature and impact on biological sequence understanding.

Findings

Deeper-layer embeddings capture more contextual knowledge.

Cross-granularity representations improve intermediate prediction tasks.

Combining models yields performance gains.

Abstract

Recent advances in general-purpose foundation models have stimulated the development of large biological sequence models. While natural language shows symbolic granularity (characters, words, sentences), biological sequences exhibit hierarchical granularity whose levels (nucleotides, amino acids, protein domains, genes) further encode biologically functional information. In this paper, we investigate the integration of cross-granularity knowledge from models through a case study of BiGCARP, a Pfam domain-level model for biosynthetic gene clusters, and ESM, an amino acid-level protein language model. Using representation analysis tools and a set of probe tasks, we first explain why a straightforward cross-model embedding initialization fails to improve downstream performance in BiGCARP, and show that deeper-layer embeddings capture a more contextual and faithful representation of the model's learned knowledge. Furthermore, we demonstrate that representations at different granularities encode complementary biological knowledge, and that combining them yields measurable performance gains in intermediate-level prediction tasks. Our findings highlight cross-granularity integration as a promising strategy for improving both the performance and interpretability of biological foundation models.