Automating IRAC Analysis in Malaysian Contract Law using a Semi-Structured Knowledge Base

TL;DR

This paper presents LegalSemi, a specialized benchmark and structured knowledge base designed to improve IRAC analysis in Malaysian Contract Law by leveraging LLMs and expert annotations.

Contribution

Introduction of LegalSemi, a novel benchmark and knowledge base tailored for legal scenario analysis and IRAC reasoning in Malaysian Contract Law.

Findings

LegalSemi enhances IRAC analysis accuracy with LLMs.

Structured knowledge base improves legal reasoning tasks.

Experiments validate the effectiveness of the approach.

Abstract

The effectiveness of Large Language Models (LLMs) in legal reasoning is often limited due to the unique legal terminologies and the necessity for highly specialized knowledge. These limitations highlight the need for high-quality data tailored for complex legal reasoning tasks. This paper introduces LegalSemi, a benchmark specifically curated for legal scenario analysis. LegalSemi comprises 54 legal scenarios, each rigorously annotated by legal experts, based on the comprehensive IRAC (Issue, Rule, Application, Conclusion) framework from Malaysian Contract Law. In addition, LegalSemi is accompanied by a structured knowledge base (SKE). A series of experiments were conducted to assess the usefulness of LegalSemi for IRAC analysis. The experimental results demonstrate the effectiveness of incorporating the SKE for issue identification, rule retrieval, application and conclusion generation using four different LLMs.