The advantages of context specific language models: the case of the Erasmian Language Model

TL;DR

This paper introduces the Erasmian Language Model, a small, context-specific model that performs well in educational settings and offers a resource-efficient alternative to large-scale models, emphasizing sustainability and privacy.

Contribution

The paper presents a novel, small-scale, context-specific language model tailored for educational use, demonstrating its effectiveness and advantages over larger models.

Findings

Performs adequately in classroom essay writing tasks

Achieves superior performance in context-specific subjects

Offers a resource-efficient alternative to large models

Abstract

The current trend to improve language model performance seems to be based on scaling up with the number of parameters (e.g. the state of the art GPT4 model has approximately 1.7 trillion parameters) or the amount of training data fed into the model. However this comes at significant costs in terms of computational resources and energy costs that compromise the sustainability of AI solutions, as well as risk relating to privacy and misuse. In this paper we present the Erasmian Language Model (ELM) a small context specific, 900 million parameter model, pre-trained and fine-tuned by and for Erasmus University Rotterdam. We show how the model performs adequately in a classroom context for essay writing, and how it achieves superior performance in subjects that are part of its context. This has implications for a wide range of institutions and organizations, showing that context specific language models may be a viable alternative for resource constrained, privacy sensitive use cases.