LegalScore: Development of a Benchmark for Evaluating AI Models in Legal Career Exams in Brazil

TL;DR

LegalScore is a new benchmark for evaluating AI models' performance in Brazilian legal career exams, emphasizing the importance of local data for accurate legal AI applications.

Contribution

The paper introduces LegalScore, a comprehensive evaluation framework for assessing AI models in Brazilian legal exams, highlighting the need for Brazil-specific training data.

Findings

Proprietary models outperform open-source ones overall.

Local models show promising performance due to Brazil-specific training.

AI still needs significant improvements to match human legal exam performance.

Abstract

This research introduces LegalScore, a specialized index for assessing how generative artificial intelligence models perform in a selected range of career exams that require a legal background in Brazil. The index evaluates fourteen different types of artificial intelligence models' performance, from proprietary to open-source models, in answering objective questions applied to these exams. The research uncovers the response of the models when applying English-trained large language models to Brazilian legal contexts, leading us to reflect on the importance and the need for Brazil-specific training data in generative artificial intelligence models. Performance analysis shows that while proprietary and most known models achieved better results overall, local and smaller models indicated promising performances due to their Brazilian context alignment in training. By establishing an evaluation framework with metrics including accuracy, confidence intervals, and normalized scoring, LegalScore enables systematic assessment of artificial intelligence performance in legal examinations in Brazil. While the study demonstrates artificial intelligence's potential value for exam preparation and question development, it concludes that significant improvements are needed before AI can match human performance in advanced legal assessments. The benchmark creates a foundation for continued research, highlighting the importance of local adaptation in artificial intelligence development.