GPT Takes the Bar Exam
Michael Bommarito II, Daniel Martin Katz

TL;DR
This study evaluates GPT-3.5's performance on the U.S. Bar Exam's multiple-choice section, showing it can pass with optimized prompts, indicating potential for AI to pass professional licensing exams.
Contribution
It demonstrates that GPT-3.5 can achieve passing scores on the Bar Exam's MBE section through prompt engineering and hyperparameter tuning, without fine-tuning.
Findings
GPT-3.5 achieves 50.3% accuracy on practice exams.
It performs at passing levels in Evidence and Torts.
Response ranking correlates strongly with correctness.
Abstract
Nearly all jurisdictions in the United States require a professional license exam, commonly referred to as "the Bar Exam," as a precondition for law practice. To even sit for the exam, most jurisdictions require that an applicant completes at least seven years of post-secondary education, including three years at an accredited law school. In addition, most test-takers also undergo weeks to months of further, exam-specific preparation. Despite this significant investment of time and capital, approximately one in five test-takers still score under the rate required to pass the exam on their first try. In the face of a complex task that requires such depth of knowledge, what, then, should we expect of the state of the art in "AI?" In this research, we document our experimental evaluation of the performance of OpenAI's `text-davinci-003` model, often-referred to as GPT-3.5, on the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsArtificial Intelligence in Law · Artificial Intelligence in Healthcare and Education · Legal Education and Practice Innovations
Methods{Dispute@FaQ-s}How to file a dispute with Expedia? · Multi-Head Attention · Attention Is All You Need · Cosine Annealing · Discriminative Fine-Tuning · Linear Warmup With Cosine Annealing · Linear Layer · Refunds@Expedia|||How do I get a full refund from Expedia? · Adam · Layer Normalization
