No Argument Left Behind: Overlapping Chunks for Faster Processing of Arbitrarily Long Legal Texts

TL;DR

This paper presents uBERT, a hybrid Transformer-RNN model designed to efficiently analyze arbitrarily long legal texts, addressing the slow processing issues in large-scale judiciary systems.

Contribution

The paper introduces uBERT, a novel hybrid model that combines Transformer and RNN architectures to handle long legal texts more efficiently than existing models.

Findings

uBERT outperforms BERT+LSTM with overlapping input.

uBERT is significantly faster than ULMFiT for long documents.

The approach maintains reasonable computational overhead.

Abstract

In a context where the Brazilian judiciary system, the largest in the world, faces a crisis due to the slow processing of millions of cases, it becomes imperative to develop efficient methods for analyzing legal texts. We introduce uBERT, a hybrid model that combines Transformer and Recurrent Neural Network architectures to effectively handle long legal texts. Our approach processes the full text regardless of its length while maintaining reasonable computational overhead. Our experiments demonstrate that uBERT achieves superior performance compared to BERT+LSTM when overlapping input is used and is significantly faster than ULMFiT for processing long legal documents.