SciFive: a text-to-text transformer model for biomedical literature

TL;DR

SciFive is a domain-specific text-to-text transformer model trained on biomedical data, outperforming existing models on various NLP tasks and highlighting the potential of text-generation methods in biomedical NLP.

Contribution

Introduces SciFive, a pre-trained biomedical T5 model that achieves state-of-the-art results across multiple NLP tasks in the biomedical domain.

Findings

SciFive outperforms BERT, BioBERT, and Base T5 on key biomedical NLP tasks.

Text-generation methods show significant potential for complex biomedical NLP tasks.

Results encourage further exploration of challenging text generation applications.

Abstract

In this report, we introduce SciFive, a domain-specific T5 model that has been pre-trained on large biomedical corpora. Our model outperforms the current SOTA methods (i.e. BERT, BioBERT, Base T5) on tasks in named entity relation, relation extraction, natural language inference, and question-answering. We show that text-generation methods have significant potential in a broad array of biomedical NLP tasks, particularly those requiring longer, more complex outputs. Our results support the exploration of more difficult text generation tasks and the development of new methods in this area