Maximizing Use-Case Specificity through Precision Model Tuning

TL;DR

This paper analyzes how fine-tuning transformer-based language models on biomedical datasets can significantly improve their relevance, accuracy, and interpretability for domain-specific information retrieval tasks, especially with smaller models.

Contribution

It provides a comparative analysis of four transformer models, highlighting the benefits of domain-specific fine-tuning and model size considerations for biomedical information retrieval.

Findings

Smaller models (<10B parameters) outperform larger models on specific biomedical questions.

Fine-tuning on domain-specific data improves relevance and interpretability.

Larger models perform better on broader prompts.

Abstract

Language models have become increasingly popular in recent years for tasks like information retrieval. As use-cases become oriented toward specific domains, fine-tuning becomes default for standard performance. To fine-tune these models for specific tasks and datasets, it is necessary to carefully tune the model's hyperparameters and training techniques. In this paper, we present an in-depth analysis of the performance of four transformer-based language models on the task of biomedical information retrieval. The models we consider are DeepMind's RETRO (7B parameters), GPT-J (6B parameters), GPT-3 (175B parameters), and BLOOM (176B parameters). We compare their performance on the basis of relevance, accuracy, and interpretability, using a large corpus of 480000 research papers on protein structure/function prediction as our dataset. Our findings suggest that smaller models, with <10B parameters and fine-tuned on domain-specific datasets, tend to outperform larger language models on highly specific questions in terms of accuracy, relevancy, and interpretability by a significant margin (+50% on average). However, larger models do provide generally better results on broader prompts.