# Large-Scale Multi-Label Text Classification on EU Legislation

## TL;DR

This paper introduces a large-scale legal text dataset and evaluates neural classifiers, demonstrating that domain-specific embeddings and selective document zones enhance multi-label classification performance, surpassing BERT in most cases.

## Contribution

The paper releases a new extensive dataset for legal multi-label classification and systematically compares neural models, highlighting the effectiveness of label-wise attention and domain-specific embeddings.

## Key findings

- BIGRUs with label-wise attention outperform other models
- Domain-specific embeddings improve classification accuracy
- Selective document zones enable bypassing BERT's length limit

## Abstract

We consider Large-Scale Multi-Label Text Classification (LMTC) in the legal domain. We release a new dataset of 57k legislative documents from EURLEX, annotated with ~4.3k EUROVOC labels, which is suitable for LMTC, few- and zero-shot learning. Experimenting with several neural classifiers, we show that BIGRUs with label-wise attention perform better than other current state of the art methods. Domain-specific WORD2VEC and context-sensitive ELMO embeddings further improve performance. We also find that considering only particular zones of the documents is sufficient. This allows us to bypass BERT's maximum text length limit and fine-tune BERT, obtaining the best results in all but zero-shot learning cases.