Legal-Tech Open Diaries: Lesson learned on how to develop and deploy light-weight models in the era of humongous Language Models

TL;DR

This paper explores developing and deploying lightweight legal language models using domain-specific training and model compression techniques, addressing high resource demands of large models in legal tech.

Contribution

It introduces domain-specific multilingual legal models and evaluates their performance, demonstrating effective compression techniques that maintain accuracy while reducing resource needs.

Findings

Domain-specific models outperform general ones in legal tasks.

Model compression techniques significantly reduce size and computational requirements.

Lightweight models achieve comparable performance to larger models.

Abstract

In the era of billion-parameter-sized Language Models (LMs), start-ups have to follow trends and adapt their technology accordingly. Nonetheless, there are open challenges since the development and deployment of large models comes with a need for high computational resources and has economical consequences. In this work, we follow the steps of the R&D group of a modern legal-tech start-up and present important insights on model development and deployment. We start from ground zero by pre-training multiple domain-specific multi-lingual LMs which are a better fit to contractual and regulatory text compared to the available alternatives (XLM-R). We present benchmark results of such models in a half-public half-private legal benchmark comprising 5 downstream tasks showing the impact of larger model size. Lastly, we examine the impact of a full-scale pipeline for model compression which includes: a) Parameter Pruning, b) Knowledge Distillation, and c) Quantization: The resulting models are much more efficient without sacrificing performance at large.