Equitable Access to Justice: Logical LLMs Show Promise

TL;DR

This paper explores integrating large language models with logic programming to improve legal reasoning, demonstrating that advanced LLMs can better encode legal texts and potentially expand access to justice.

Contribution

It introduces a method combining LLMs with logic programming to enhance legal reasoning capabilities, focusing on insurance contracts.

Findings

GPT-4o fails to encode legal contracts into logic

OpenAI o1-preview successfully encodes a health insurance contract

Advanced LLMs with System 2 reasoning can improve legal text encoding

Abstract

The costs and complexity of the American judicial system limit access to legal solutions for many Americans. Large language models (LLMs) hold great potential to improve access to justice. However, a major challenge in applying AI and LLMs in legal contexts, where consistency and reliability are crucial, is the need for System 2 reasoning. In this paper, we explore the integration of LLMs with logic programming to enhance their ability to reason, bringing their strategic capabilities closer to that of a skilled lawyer. Our objective is to translate laws and contracts into logic programs that can be applied to specific legal cases, with a focus on insurance contracts. We demonstrate that while GPT-4o fails to encode a simple health insurance contract into logical code, the recently released OpenAI o1-preview model succeeds, exemplifying how LLMs with advanced System 2 reasoning capabilities can expand access to justice.