Parameter-Efficient Legal Domain Adaptation

TL;DR

This paper introduces a parameter-efficient legal domain adaptation method that uses unsupervised legal data to achieve strong performance on legal tasks while tuning only a tiny fraction of model parameters.

Contribution

It presents a novel approach for legal domain adaptation that significantly reduces parameter tuning requirements using large-scale unsupervised legal data.

Findings

Outperforms or matches existing models like LEGAL-BERT in few-shot settings

Tunes only about 0.1% of model parameters

Achieves calibration comparable to existing methods

Abstract

Seeking legal advice is often expensive. Recent advancements in machine learning for solving complex problems can be leveraged to help make legal services more accessible to the public. However, real-life applications encounter significant challenges. State-of-the-art language models are growing increasingly large, making parameter-efficient learning increasingly important. Unfortunately, parameter-efficient methods perform poorly with small amounts of data, which are common in the legal domain (where data labelling costs are high). To address these challenges, we propose parameter-efficient legal domain adaptation, which uses vast unsupervised legal data from public legal forums to perform legal pre-training. This method exceeds or matches the fewshot performance of existing models such as LEGAL-BERT on various legal tasks while tuning only approximately 0.1% of model parameters. Additionally, we show that our method can achieve calibration comparable to existing methods across several tasks. To the best of our knowledge, this work is among the first to explore parameter-efficient methods of tuning language models in the legal domain.