LLaMA-Gene: A General-purpose Gene Task Large Language Model Based on Instruction Fine-tuning

TL;DR

This paper expands the LLaMA large language model to handle gene sequences by incorporating specialized tokenization and instruction fine-tuning, enabling it to perform various gene-related tasks effectively.

Contribution

The authors develop a unified model that integrates natural language and gene sequences through vocabulary expansion and instruction fine-tuning, achieving competitive results in gene tasks.

Findings

Achieved SOTA-like performance in gene classification.

Successfully integrated gene and natural language processing.

Demonstrated the feasibility of a unified gene-language model.

Abstract

Building a general-purpose task model similar to ChatGPT has been an important research direction for gene large language models. Instruction fine-tuning is a key component in building ChatGPT, but existing instructions are primarily based on natural language. Natural language and gene sequences have significant differences in tokenization and encoding. Therefore, constructing a multilingual model that can handle both natural language and gene sequences is crucial for solving this problem.In this paper, we expand the capabilities of the LLaMA large language model to include gene language. This involves expanding the vocabulary using the Byte Pair Encoding (BPE) method, specifically tailored for DNA and protein sequences, and conducting further pre-training on these sequences. We then convert various downstream gene task data into a unified format for instruction fine-tuning and further fine-tune the model on this data.Our study demonstrates that a mixed model of gene and natural language, fine-tuned with instructions, achieves results comparable to the current state-of-the-art (SOTA) in tasks such as gene classification and gene sequence interaction. This provides a promising direction for building a unified large language model for gene tasks.