Bringing order into the realm of Transformer-based language models for artificial intelligence and law

TL;DR

This paper systematically reviews how Transformer-based language models have advanced AI applications in the legal domain, highlighting progress, limitations, and future research opportunities.

Contribution

It provides the first comprehensive overview of TLM methods in legal AI, analyzing their impact and identifying research gaps.

Findings

Transformers significantly improved legal NLP tasks

Current models face limitations in legal reasoning

Opportunities exist for enhancing model interpretability

Abstract

Transformer-based language models (TLMs) have widely been recognized to be a cutting-edge technology for the successful development of deep-learning-based solutions to problems and applications that require natural language processing and understanding. Like for other textual domains, TLMs have indeed pushed the state-of-the-art of AI approaches for many tasks of interest in the legal domain. Despite the first Transformer model being proposed about six years ago, there has been a rapid progress of this technology at an unprecedented rate, whereby BERT and related models represent a major reference, also in the legal domain. This article provides the first systematic overview of TLM-based methods for AI-driven problems and tasks in the legal sphere. A major goal is to highlight research advances in this field so as to understand, on the one hand, how the Transformers have contributed to the success of AI in supporting legal processes, and on the other hand, what are the current limitations and opportunities for further research development.