Considerations for health care institutions training large language models on electronic health records

TL;DR

This paper analyzes the considerations, costs, and strategies for healthcare institutions to train or fine-tune large language models on electronic health records, focusing on data size, model size, and budget constraints.

Contribution

It provides a framework for understanding the trade-offs in training LLMs on EHR data, addressing cost, data, and model size considerations.

Findings

Dataset size impacts model performance and training costs.

Fine-tuning open-source models can be cost-effective for healthcare applications.

Training from scratch requires substantial data and resources.

Abstract

Large language models (LLMs) like ChatGPT have excited scientists across fields; in medicine, one source of excitement is the potential applications of LLMs trained on electronic health record (EHR) data. But there are tough questions we must first answer if health care institutions are interested in having LLMs trained on their own data; should they train an LLM from scratch or fine-tune it from an open-source model? For healthcare institutions with a predefined budget, what are the biggest LLMs they can afford? In this study, we take steps towards answering these questions with an analysis on dataset sizes, model sizes, and costs for LLM training using EHR data. This analysis provides a framework for thinking about these questions in terms of data scale, compute scale, and training budgets.