Extracting Norms from Contracts Via ChatGPT: Opportunities and Challenges

TL;DR

This paper evaluates ChatGPT's ability to extract norms from contracts, highlighting its strengths in doing so without training but also its limitations like oversight and hallucination, which impact accuracy.

Contribution

It demonstrates ChatGPT's potential for extracting norms from contracts without fine-tuning and discusses its limitations, informing future improvements in automated norm extraction.

Findings

ChatGPT can extract norms without training or fine-tuning.

Limitations include oversight, hallucination, and parsing errors.

Enhanced extraction can improve multiagent system transparency.

Abstract

We investigate the effectiveness of ChatGPT in extracting norms from contracts. Norms provide a natural way to engineer multiagent systems by capturing how to govern the interactions between two or more autonomous parties. We extract norms of commitment, prohibition, authorization, and power, along with associated norm elements (the parties involved, antecedents, and consequents) from contracts. Our investigation reveals ChatGPT's effectiveness and limitations in norm extraction from contracts. ChatGPT demonstrates promising performance in norm extraction without requiring training or fine-tuning, thus obviating the need for annotated data, which is not generally available in this domain. However, we found some limitations of ChatGPT in extracting these norms that lead to incorrect norm extractions. The limitations include oversight of crucial details, hallucination, incorrect parsing of conjunctions, and empty norm elements. Enhanced norm extraction from contracts can foster the development of more transparent and trustworthy formal agent interaction specifications, thereby contributing to the improvement of multiagent systems.