A Small Claims Court for the NLP: Judging Legal Text Classification Strategies With Small Datasets

TL;DR

This study evaluates various NLP strategies for legal text classification with limited labeled data, finding that semi-supervised learning with BERT and data augmentation yields the best accuracy in a low-resource legal domain.

Contribution

It demonstrates the effectiveness of semi-supervised learning and data augmentation techniques, particularly UDA with BERT, for legal text classification with small datasets.

Findings

Unsupervised Data Augmentation (UDA) achieved 80.7% accuracy.

Classical models like SVM and logistic regression performed well with word2vec embeddings.

BERT combined with semi-supervised strategies outperformed other models.

Abstract

Recent advances in language modelling has significantly decreased the need of labelled data in text classification tasks. Transformer-based models, pre-trained on unlabeled data, can outmatch the performance of models trained from scratch for each task. However, the amount of labelled data need to fine-tune such type of model is still considerably high for domains requiring expert-level annotators, like the legal domain. This paper investigates the best strategies for optimizing the use of a small labeled dataset and large amounts of unlabeled data and perform a classification task in the legal area with 50 predefined topics. More specifically, we use the records of demands to a Brazilian Public Prosecutor's Office aiming to assign the descriptions in one of the subjects, which currently demands deep legal knowledge for manual filling. The task of optimizing the performance of classifiers in this scenario is especially challenging, given the low amount of resources available regarding the Portuguese language, especially in the legal domain. Our results demonstrate that classic supervised models such as logistic regression and SVM and the ensembles random forest and gradient boosting achieve better performance along with embeddings extracted with word2vec when compared to BERT language model. The latter demonstrates superior performance in association with the architecture of the model itself as a classifier, having surpassed all previous models in that regard. The best result was obtained with Unsupervised Data Augmentation (UDA), which jointly uses BERT, data augmentation, and strategies of semi-supervised learning, with an accuracy of 80.7% in the aforementioned task.