Long-Context Long-Form Question Answering for Legal Domain

TL;DR

This paper presents a specialized long-context question answering system for legal documents, capable of handling complex layouts, domain-specific language, and providing comprehensive long-form answers, with a new coverage metric for evaluation.

Contribution

It introduces a novel legal QA system that deconstructs vocabulary, parses complex layouts, and generates detailed answers, along with a curated legal QA dataset and a coverage evaluation metric.

Findings

The system effectively handles complex legal document structures.

It improves retrieval and comprehension of legal information.

Experimental results demonstrate the system's usability and effectiveness.

Abstract

Legal documents have complex document layouts involving multiple nested sections, lengthy footnotes and further use specialized linguistic devices like intricate syntax and domain-specific vocabulary to ensure precision and authority. These inherent characteristics of legal documents make question answering challenging, and particularly so when the answer to the question spans several pages (i.e. requires long-context) and is required to be comprehensive (i.e. a long-form answer). In this paper, we address the challenges of long-context question answering in context of long-form answers given the idiosyncrasies of legal documents. We propose a question answering system that can (a) deconstruct domain-specific vocabulary for better retrieval from source documents, (b) parse complex document layouts while isolating sections and footnotes and linking them appropriately, (c) generate comprehensive answers using precise domain-specific vocabulary. We also introduce a coverage metric that classifies the performance into recall-based coverage categories allowing human users to evaluate the recall with ease. We curate a QA dataset by leveraging the expertise of professionals from fields such as law and corporate tax. Through comprehensive experiments and ablation studies, we demonstrate the usability and merit of the proposed system.