Passing the Brazilian OAB Exam: data preparation and some experiments
Pedro Delfino, Bruno Cuconato, Edward Hermann Haeusler and, Alexandre Rademaker
TL;DR
This paper introduces a new dataset for the Brazilian OAB legal exam and presents initial experiments on legal reasoning, establishing a baseline for future research in legal AI systems.
Contribution
It constructs a novel dataset for the Brazilian OAB exam and provides preliminary experiments to evaluate legal reasoning capabilities.
Findings
Baseline performance established for legal question justification
Identifies challenges in current legal reasoning models
Proposes directions for improving legal AI systems
Abstract
In Brazil, all legal professionals must demonstrate their knowledge of the law and its application by passing the OAB exams, the national bar exams. The OAB exams therefore provide an excellent benchmark for the performance of legal information systems since passing the exam would arguably signal that the system has acquired capacity of legal reasoning comparable to that of a human lawyer. This article describes the construction of a new data set and some preliminary experiments on it, treating the problem of finding the justification for the answers to questions. The results provide a baseline performance measure against which to evaluate future improvements. We discuss the reasons to the poor performance and propose next steps.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsArtificial Intelligence in Law · Imbalanced Data Classification Techniques · Data Quality and Management