Juru: Legal Brazilian Large Language Model from Reputable Sources

TL;DR

This paper presents Juru, a domain-specialized large language model for Brazilian legal texts, demonstrating improved legal performance but increased forgetting in general knowledge, highlighting data quality's role in model training.

Contribution

It introduces Juru, a legal domain-specific LLM trained on reputable Brazilian legal sources, showing benefits and trade-offs of domain specialization with high-quality data.

Findings

Juru outperforms general models on legal benchmarks.

Domain specialization improves legal knowledge understanding.

Specialization increases forgetting in unrelated domains.

Abstract

The high compute cost associated with pretraining large language models limits their research. Two strategies have emerged to address this issue: domain specialization and pretraining with high-quality data. To explore these strategies, we specialized the Mistral-7B model with 1.9 billion unique tokens from reputable Brazilian legal sources and conducted few-shot evaluations on legal and general knowledge test suites. Our model, Juru, demonstrates the benefits of domain specialization by achieving improved performance on legal benchmarks, even with a reduced amount of pretraining data. However, this domain specialization through continued pretraining comes at the cost of increased forgetting in unrelated domains, as evidenced by performance degradation on general knowledge test suites in both Portuguese and English. This study contributes to the growing body of scientific evidence showing that pretraining data selection may enhance the performance of large language models, enabling the exploration of these models at a lower cost. Juru is publicly available at https://huggingface.co/roseval/Juru-7B .