Sequence-aware multimodal page classification of Brazilian legal documents

TL;DR

This paper presents a multimodal sequence-aware classification method for Brazilian legal documents, combining visual and textual features with sequence modeling to improve accuracy in classifying court case pages.

Contribution

It introduces a novel multimodal dataset and a fusion approach that effectively integrates visual and textual features, handling missing data and leveraging page sequence information.

Findings

Multimodal classifiers outperform unimodal ones in accuracy.

Sequence modeling enhances classification performance.

The fusion module effectively handles missing visual or textual data.

Abstract

The Brazilian Supreme Court receives tens of thousands of cases each semester. Court employees spend thousands of hours to execute the initial analysis and classification of those cases -- which takes effort away from posterior, more complex stages of the case management workflow. In this paper, we explore multimodal classification of documents from Brazil's Supreme Court. We train and evaluate our methods on a novel multimodal dataset of 6,510 lawsuits (339,478 pages) with manual annotation assigning each page to one of six classes. Each lawsuit is an ordered sequence of pages, which are stored both as an image and as a corresponding text extracted through optical character recognition. We first train two unimodal classifiers: a ResNet pre-trained on ImageNet is fine-tuned on the images, and a convolutional network with filters of multiple kernel sizes is trained from scratch on document texts. We use them as extractors of visual and textual features, which are then combined through our proposed Fusion Module. Our Fusion Module can handle missing textual or visual input by using learned embeddings for missing data. Moreover, we experiment with bi-directional Long Short-Term Memory (biLSTM) networks and linear-chain conditional random fields to model the sequential nature of the pages. The multimodal approaches outperform both textual and visual classifiers, especially when leveraging the sequential nature of the pages.