Na Pr\'atica, qual IA Entende o Direito? Um Estudo Experimental com IAs Generalistas e uma IA Jur\'idica

TL;DR

This paper evaluates the legal understanding of general-purpose AI systems versus a specialized legal AI through an experimental protocol involving legal professionals, highlighting the importance of domain-specific training for reliable legal AI performance.

Contribution

It introduces an evaluation protocol combining legal theory and empirical assessment, demonstrating the superiority of a domain-specific AI over general-purpose models in legal tasks.

Findings

JusIA outperformed general-purpose AIs in legal tasks

Domain specialization improves legal AI reliability

The evaluation protocol effectively assesses legal AI systems

Abstract

This study presents the Jusbrasil Study on the Use of General-Purpose AIs in Law, proposing an experimental evaluation protocol combining legal theory, such as material correctness, systematic coherence, and argumentative integrity, with empirical assessment by 48 legal professionals. Four systems (JusIA, ChatGPT Free, ChatGPT Pro, and Gemini) were tested in tasks simulating lawyers' daily work. JusIA, a domain-specialized model, consistently outperformed the general-purpose systems, showing that both domain specialization and a theoretically grounded evaluation are essential for reliable legal AI outputs.