A Prompt Engineering Approach and a Knowledge Graph based Framework for Tackling Legal Implications of Large Language Model Answers

TL;DR

This paper analyzes the legal risks of LLMs, proposes prompt engineering to mitigate issues, and introduces a knowledge graph framework to generate legal citations, aiming to improve trustworthiness and legal compliance.

Contribution

It presents a novel prompt re-engineering approach to isolate legal issues and a legal knowledge graph framework for generating citations in LLM responses.

Findings

Prompt engineering can partially isolate legal issues in LLM outputs.

Legal knowledge graph enhances response credibility with citations.

Limitations exist in current prompt-based solutions.

Abstract

With the recent surge in popularity of Large Language Models (LLMs), there is the rising risk of users blindly trusting the information in the response, even in cases where the LLM recommends actions that have potential legal implications and this may put the user in danger. We provide an empirical analysis on multiple existing LLMs showing the urgency of the problem. Hence, we propose a short-term solution consisting in an approach for isolating these legal issues through prompt re-engineering. We further analyse the outcomes but also the limitations of the prompt engineering based approach and we highlight the need of additional resources for fully solving the problem We also propose a framework powered by a legal knowledge graph (KG) to generate legal citations for these legal issues, enriching the response of the LLM.