TiME: Tiny Monolingual Encoders for Efficient NLP Pipelines

TL;DR

TiME introduces small, efficient monolingual NLP models trained with distillation, achieving a better balance of performance, speed, and energy use for resource-constrained applications, including support for low-resource languages.

Contribution

The paper presents TiME, a novel approach to training tiny monolingual encoders that are efficient and effective, using modern distillation techniques and supporting low-resource languages.

Findings

TiME models outperform larger models in speed and energy efficiency.

Distillation from multilingual teachers is feasible for monolingual models.

Models with absolute positional embeddings can be distilled from relative positional embedding teachers.

Abstract

Today, a lot of research on language models is focused on large, general-purpose models. However, many NLP pipelines only require models with a well-defined, small set of capabilities. While large models are capable of performing the tasks of those smaller models, they are simply not fast enough to process large amounts of data or offer real-time responses. Furthermore, they often use unnecessarily large amounts of energy, leading to sustainability concerns and problems when deploying them on battery-powered devices. In our work, we show how to train small models for such efficiency-critical applications. As opposed to many off-the-shelf NLP pipelines, our models use modern training techniques such as distillation, and offer support for low-resource languages. We call our models TiME (Tiny Monolingual Encoders) and comprehensively evaluate them on a range of common NLP tasks, observing an improved trade-off between benchmark performance on one hand, and throughput, latency and energy consumption on the other. Along the way, we show that distilling monolingual models from multilingual teachers is possible, and likewise distilling models with absolute positional embeddings from teachers with relative positional embeddings.