Juru: Legal Brazilian Large Language Model from Reputable Sources
Roseval Malaquias Junior, Ramon Pires, Roseli Romero, Rodrigo Nogueira

TL;DR
This paper presents Juru, a domain-specialized large language model for Brazilian legal texts, demonstrating improved legal performance but increased forgetting in general knowledge, highlighting data quality's role in model training.
Contribution
It introduces Juru, a legal domain-specific LLM trained on reputable Brazilian legal sources, showing benefits and trade-offs of domain specialization with high-quality data.
Findings
Juru outperforms general models on legal benchmarks.
Domain specialization improves legal knowledge understanding.
Specialization increases forgetting in unrelated domains.
Abstract
The high compute cost associated with pretraining large language models limits their research. Two strategies have emerged to address this issue: domain specialization and pretraining with high-quality data. To explore these strategies, we specialized the Mistral-7B model with 1.9 billion unique tokens from reputable Brazilian legal sources and conducted few-shot evaluations on legal and general knowledge test suites. Our model, Juru, demonstrates the benefits of domain specialization by achieving improved performance on legal benchmarks, even with a reduced amount of pretraining data. However, this domain specialization through continued pretraining comes at the cost of increased forgetting in unrelated domains, as evidenced by performance degradation on general knowledge test suites in both Portuguese and English. This study contributes to the growing body of scientific evidence…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques · Artificial Intelligence in Law · Legal Language and Interpretation
