Large-Scale Multi-Label Text Classification on EU Legislation

TL;DR

This paper introduces a large-scale legal text dataset and evaluates neural classifiers, demonstrating that domain-specific embeddings and selective document zones enhance multi-label classification performance, surpassing BERT in most cases.

Contribution

The paper releases a new extensive dataset for legal multi-label classification and systematically compares neural models, highlighting the effectiveness of label-wise attention and domain-specific embeddings.

Findings

BIGRUs with label-wise attention outperform other models

Domain-specific embeddings improve classification accuracy

Selective document zones enable bypassing BERT's length limit

Abstract

We consider Large-Scale Multi-Label Text Classification (LMTC) in the legal domain. We release a new dataset of 57k legislative documents from EURLEX, annotated with ~4.3k EUROVOC labels, which is suitable for LMTC, few- and zero-shot learning. Experimenting with several neural classifiers, we show that BIGRUs with label-wise attention perform better than other current state of the art methods. Domain-specific WORD2VEC and context-sensitive ELMO embeddings further improve performance. We also find that considering only particular zones of the documents is sufficient. This allows us to bypass BERT's maximum text length limit and fine-tune BERT, obtaining the best results in all but zero-shot learning cases.

Tables

Table 1. Table 1: Statistics of the eur-lex dataset.

Subset	Documents ($D$)	Words/$D$	Labels/$D$
Train	45,000	729	5
Dev.	6,000	714	5
Test	6,000	725	5
Total	57,000	727	5

Table 2. Table 2: Results on eurlex57k for all, frequent, few-shot, zero-shot labels. Starred methods use the first 512 document tokens; all other methods use full documents. Unless otherwise stated, glove embeddings are used.

	All Labels			Frequent		Few		Zero
	$RP@5$	$nDCG@5$	Micro-$F1$	$RP@5$	$nDCG@5$	$RP@5$	$nDCG@5$	$RP@5$	$nDCG@5$
Exact Match	0.097	0.099	0.120	0.219	0.201	0.111	0.074	0.194	0.186
Logistic Regression	0.710	0.741	0.539	0.767	0.781	0.508	0.470	0.011	0.011
bigru-att	0.758	0.789	0.689	0.799	0.813	0.631	0.580	0.040	0.027
han	0.746	0.778	0.680	0.789	0.805	0.597	0.544	0.051	0.034
cnn-lwan	0.716	0.746	0.642	0.761	0.772	0.613	0.557	0.036	0.023
bigru-lwan	0.766	0.796	0.698	0.805	0.819	0.662	0.618	0.029	0.019
zero-cnn-lwan	0.684	0.717	0.618	0.730	0.745	0.495	0.454	0.321	0.264
zero-bigru-lwan	0.718	0.752	0.652	0.764	0.780	0.561	0.510	0.438	0.345
bigru-lwan-l2v	0.775	0.804	0.711	0.815	0.828	0.656	0.612	0.034	0.024
bigru-lwan-l2v*	0.770	0.796	0.709	0.811	0.825	0.641	0.600	0.047	0.030
bigru-lwan-elmo*	0.781	0.811	0.719	0.821	0.835	0.668	0.619	0.044	0.028
bert-base *	0.796	0.823	0.732	0.835	0.846	0.686	0.636	0.028	0.023

Table 3. Table 3: bigru-lwan with glove , l2v , elmo .

	$RP@5$	$nDCG@5$	Micro-$F1$
glove	0.766	0.796	0.698
l2v	0.775	0.804	0.711
glove + elmo	0.777	0.808	0.714
l2v + elmo	0.781	0.811	0.719

Table 4. Table 4: bigru-lwan with different document zones.

	${\mu }_{words}$	$RP@5$	$nDCG@5$	Micro-$F1$
H	43	0.747	0.782	0.688
R	317	0.734	0.765	0.669
H+R	360	0.765	0.796	0.701
MB	187	0.643	0.674	0.590
Full	727	0.766	0.797	0.702

Table 5. Table 5: Best hyper parameters for neural methods. N l subscript 𝑁 𝑙 N_{l} : number of layers, H ​ U 𝐻 𝑈 HU : hidden units size, D d subscript 𝐷 𝑑 D_{d} : dropout rate across dimensions, D w ​ e subscript 𝐷 𝑤 𝑒 D_{we} : dropout rate of word embeddings, B ​ S 𝐵 𝑆 BS : batch size. * Hidden units size is fixed to word embedding dimensionality, + N l subscript 𝑁 𝑙 N_{l} , H ​ U 𝐻 𝑈 HU are fixed from the pre-trained model. Dropout rate fixed as suggested by Devlin et al. (2018).

Hyper parameters	bigru-att	han	cnn-lwan	bigru-lwan	zacnn *	zagru *	bert-base +
$N_l\in [1,2]$	1	(1,1)	1	1	1	1	12
$HU\in [200,300,400]$	300	(300,300)	200	300	200	100	768
$D_d\in [0.1,0.2,\dots ,0.5]$	0.2	0.3	0.1	0.4	0.1	0.1	0.1
$D_{we}\in [0.00,0.01,0.02]$	0.02	0.02	0.01	0.00	0.00	0.00	0.00
$BS\in [8,12,16]$	12	16	12	16	16	16	8

Table 6. Table 6: P ​ @ ​ 1 𝑃 @ 1 P@1 , P ​ @ ​ 5 𝑃 @ 5 P@5 and P ​ @ ​ 10 𝑃 @ 10 P@10 results on eurlex57k for all, frequent, few-shot, zero-shot labels. Starred methods use the first 512 document tokens; all other methods use full documents. Unless otherwise stated, glove embeddings are used.

	Overall			Frequent			Few			Zero
	@1	@5	@10	@1	@5	@10	@1	@5	@10	@1	@5	@10
Exact Match	0.131	0.084	0.080	0.194	0.166	0.141	0.037	0.037	0.036	0.178	0.042	0.022
Logistic Regression	0.861	0.613	0.378	0.864	0.604	0.368	0.458	0.169	0.094	0.011	0.002	0.002
bigru-att	0.899	0.654	0.407	0.893	0.627	0.382	0.551	0.212	0.121	0.015	0.008	0.007
han	0.894	0.643	0.401	0.889	0.620	0.378	0.510	0.199	0.114	0.020	0.011	0.008
cnn-lwan	0.853	0.617	0.395	0.849	0.596	0.374	0.521	0.204	0.117	0.011	0.007	0.007
bigru-lwan	0.907	0.661	0.414	0.900	0.631	0.387	0.599	0.222	0.124	0.011	0.006	0.006
zero-cnn-lwan	0.842	0.589	0.371	0.837	0.572	0.355	0.447	0.164	0.094	0.202	0.069	0.040
zero-bigru-lwan	0.874	0.619	0.386	0.867	0.599	0.367	0.488	0.184	0.107	0.247	0.093	0.057
bigru-lwan-l2v	0.913	0.669	0.417	0.905	0.639	0.390	0.593	0.219	0.122	0.013	0.007	0.008
bigru-lwan-l2v*	0.915	0.664	0.413	0.905	0.637	0.387	0.586	0.214	0.120	0.013	0.010	0.010
bigru-lwan-elmo*	0.921	0.674	0.419	0.912	0.644	0.391	0.595	0.226	0.127	0.011	0.009	0.007
bert-base *	0.922	0.687	0.424	0.914	0.656	0.394	0.611	0.229	0.129	0.019	0.006	0.007

Table 7. Table 7: R ​ @ ​ 1 𝑅 @ 1 R@1 , R ​ @ ​ 5 𝑅 @ 5 R@5 and R ​ @ ​ 10 𝑅 @ 10 R@10 results on eurlex57k for all, frequent, few-shot, zero-shot labels. Starred methods use the first 512 document tokens; all other methods use full documents. Unless otherwise stated, glove embeddings are used.

	Overall			Frequent			Few			Zero
	@1	@5	@10	@1	@5	@10	@1	@5	@10	@1	@5	@10
Exact Match	0.026	0.087	0.168	0.045	0.207	0.344	0.022	0.111	0.214	0.161	0.194	0.206
Logistic Regression	0.195	0.641	0.764	0.234	0.719	0.845	0.313	0.507	0.560	0.011	0.011	0.022
bigru-att	0.204	0.685	0.824	0.242	0.749	0.880	0.382	0.629	0.703	0.015	0.040	0.062
han	0.203	0.675	0.811	0.241	0.740	0.871	0.355	0.596	0.673	0.018	0.051	0.079
cnn-lwan	0.193	0.647	0.800	0.229	0.713	0.862	0.360	0.612	0.681	0.011	0.036	0.061
bigru-lwan	0.205	0.692	0.836	0.243	0.755	0.891	0.420	0.661	0.725	0.011	0.029	0.060
zero-cnn-lwan	0.189	0.617	0.752	0.223	0.683	0.820	0.300	0.494	0.556	0.189	0.321	0.376
zero-bigru-lwan	0.197	0.648	0.782	0.232	0.716	0.847	0.335	0.560	0.635	0.231	0.438	0.531
bigru-lwan-l2v	0.207	0.700	0.842	0.246	0.764	0.898	0.414	0.655	0.716	0.012	0.034	0.066
bigru-lwan-l2v*	0.207	0.696	0.835	0.245	0.760	0.891	0.409	0.640	0.707	0.013	0.047	0.084
bigru-lwan-elmo*	0.208	0.705	0.844	0.249	0.770	0.900	0.410	0.667	0.732	0.011	0.044	0.061
bert-base *	0.209	0.719	0.855	0.250	0.784	0.908	0.428	0.684	0.752	0.018	0.028	0.068

Table 8. Table 8: R ​ P ​ @ ​ 1 𝑅 𝑃 @ 1 RP@1 , R ​ P ​ @ ​ 5 𝑅 𝑃 @ 5 RP@5 and R ​ P ​ @ ​ 10 𝑅 𝑃 @ 10 RP@10 results on eurlex57k for all, frequent, few-shot, zero-shot labels. Starred methods use the first 512 document tokens; all other methods use full documents. Unless otherwise stated, glove embeddings are used.

	Overall			Frequent			Few			Zero
	@1	@5	@10	@1	@5	@10	@1	@5	@10	@1	@5	@10
Exact Match	0.131	0.097	0.168	0.194	0.219	0.344	0.037	0.111	0.214	0.178	0.194	0.206
Logistic Regression	0.861	0.710	0.765	0.864	0.767	0.846	0.458	0.508	0.560	0.011	0.011	0.022
bigru-att	0.899	0.758	0.824	0.893	0.799	0.880	0.551	0.631	0.703	0.015	0.040	0.062
hans	0.894	0.746	0.811	0.889	0.789	0.872	0.510	0.597	0.673	0.020	0.051	0.079
cnn-lwan	0.853	0.716	0.801	0.849	0.761	0.862	0.521	0.613	0.681	0.011	0.036	0.061
bigru-lwan	0.907	0.766	0.836	0.900	0.805	0.891	0.599	0.662	0.725	0.011	0.029	0.060
zero-cnn-lwan	0.842	0.684	0.753	0.837	0.730	0.820	0.447	0.495	0.556	0.202	0.321	0.376
zero-bigru-lwan	0.874	0.718	0.782	0.867	0.764	0.847	0.488	0.561	0.635	0.247	0.438	0.531
bigru-lwan-l2v	0.913	0.775	0.842	0.905	0.815	0.898	0.593	0.657	0.716	0.013	0.034	0.066
bigru-lwan-l2v*	0.915	0.770	0.836	0.905	0.811	0.891	0.586	0.641	0.707	0.013	0.047	0.084
bigru-lwan-elmo*	0.921	0.781	0.845	0.912	0.821	0.901	0.595	0.668	0.732	0.011	0.044	0.061
bert-base *	0.922	0.796	0.856	0.914	0.835	0.908	0.611	0.686	0.752	0.019	0.028	0.068

Table 9. Table 9: n ​ D ​ C ​ G ​ @ ​ 1 𝑛 𝐷 𝐶 𝐺 @ 1 nDCG@1 , n ​ D ​ C ​ G ​ @ ​ 5 𝑛 𝐷 𝐶 𝐺 @ 5 nDCG@5 and n ​ D ​ C ​ G ​ @ ​ 10 𝑛 𝐷 𝐶 𝐺 @ 10 nDCG@10 results on eurlex57k for all, frequent, few-shot, zero-shot labels. Starred methods use the first 512 document tokens; all other methods use full documents. Unless otherwise stated, glove embeddings are used.

	Overall			Frequent			Few			Zero
	@1	@5	@10	@1	@5	@10	@1	@5	@10	@1	@5	@10
Exact Match	0.131	0.099	0.134	0.194	0.201	0.262	0.037	0.074	0.112	0.178	0.186	0.189
Logistic Regression	0.861	0.741	0.766	0.864	0.781	0.819	0.458	0.470	0.489	0.011	0.011	0.014
bigru-att	0.899	0.789	0.819	0.893	0.813	0.853	0.551	0.580	0.608	0.015	0.027	0.034
han	0.894	0.778	0.808	0.889	0.805	0.845	0.510	0.544	0.573	0.020	0.034	0.043
cnn-lwan	0.853	0.746	0.786	0.849	0.772	0.822	0.521	0.557	0.583	0.011	0.023	0.032
bigru-lwan	0.907	0.796	0.829	0.900	0.819	0.861	0.599	0.618	0.643	0.011	0.019	0.029
zero-cnn-lwan	0.842	0.717	0.749	0.837	0.745	0.789	0.447	0.454	0.478	0.202	0.264	0.281
zero-bigru-lwan	0.874	0.752	0.781	0.867	0.780	0.819	0.488	0.510	0.539	0.247	0.345	0.375
bigru-lwan-l2v	0.913	0.804	0.836	0.905	0.828	0.869	0.593	0.612	0.635	0.013	0.024	0.035
bigru-lwan-l2v*	0.915	0.801	0.832	0.905	0.825	0.864	0.586	0.600	0.625	0.013	0.030	0.042
bigru-lwan-elmo*	0.921	0.811	0.841	0.912	0.835	0.874	0.595	0.619	0.643	0.011	0.028	0.034
bert-base *	0.922	0.823	0.851	0.914	0.846	0.882	0.611	0.636	0.662	0.019	0.023	0.036

Equations

$$R@K$$

$$P@K$$

$$RP@K$$

$$nDCG@K$$

Full text

1 Introduction

Large-scale multi-label text classification (lmtc) is the task of assigning to each document all the relevant labels from a large set, typically containing thousands of labels (classes). Applications include building web directories (Partalas et al., 2015), labeling scientific publications with concepts from ontologies Tsatsaronis et al. (2015), assigning diagnostic and procedure labels to medical records Mullenbach et al. (2018); Rios and Kavuluru (2018). We focus on legal text processing, an emerging nlp field with many applications (e.g., legal judgment Nallapati and Manning (2008); Aletras et al. (2016), contract element extraction Chalkidis et al. (2017), obligation extraction Chalkidis et al. (2018)), but limited publicly available resources.

Our first contribution is a new publicly available legal lmtc dataset, dubbed eurlex57k, containing 57k English eu legislative documents from the eur-lex portal, tagged with $\mathtt{\sim}$4.3k labels (concepts) from the European Vocabulary (eurovoc).111See https://eur-lex.europa.eu/ for eur-lex, and https://publications.europa.eu/en/web/eu-vocabularies for eurovoc. eurovoc contains approx. 7k labels, but most of them are rarely used, hence they are under-represented (or absent) in eurlex57k, making the dataset also appropriate for few- and zero-shot learning. eurlex57k can be viewed as an improved version of the dataset released by Mencia and Fürnkranzand (2007), which has been widely used in lmtc research, but is less than half the size of eurlex57k (19.6k documents, 4k eurovoc labels) and more than ten years old.

As a second contribution, we experiment with several neural classifiers on eurlex57k, including the Label-Wise Attention Network of Mullenbach et al. (2018), called cnn-lwan here, which was reported to achieve state of the art performance in lmtc on medical records. We show that a simpler bigru with self-attention Xu et al. (2015) outperforms cnn-lwan by a wide margin on eurlex57k. However, by replacing the cnn encoder of cnn-lwan with a bigru, we obtain even better results on eurlex57k. Domain-specific word2vec Mikolov et al. (2013) and context-sensitive elmo embeddings Peters et al. (2018) yield further improvements. We thus establish strong baselines for eurlex57k.

As a third contribution, we investigate which zones of the documents are more informative on eurlex57k, showing that considering only the title and recitals of each document leads to almost the same performance as considering the full document. This allows us to bypass bert’s Devlin et al. (2018) maximum text length limit and fine-tune bert, obtaining the best results for all but zero-shot learning labels. To our knowledge, this is the first application of bert to an lmtc task, which provides further evidence of the superiority of pretrained language models with task-specific fine-tuning, and establishes an even stronger baseline for eurlex57k and lmtc in general.

2 Related Work

You et al. You et al. (2018) explored rnn-based methods with self-attention on five lmtc datasets that had also been considered by Liu et al. (2017), namely rcv1 Lewis et al. (2004), Amazon-13K, McAuley and Leskovec (2013), Wiki-30K and Wiki-500K Zubiaga (2012), as well as the previous eur-lex dataset Mencia and Fürnkranzand (2007), reporting that attention-based rnns produced the best results overall (4 out of 5 datasets).

Mullenbach et al. (2018) investigated the use of label-wise attention in lmtc for medical code prediction on the mimic-ii and mimic-iii datasets Johnson et al. (2017). Their best method, Convolutional Attention for Multi-Label Classification, called cnn-lwan here, employs one attention head per label and was shown to outperform weak baselines, namely logistic regression, plain bigrus, cnns with a single convolution layer.

Rios and Kavuluru (2018) consider few- and zero-shot learning on the mimic datasets. They propose Zero-shot Attentive cnn, called zero-cnn-lwan here, a method similar to cnn-lwan, which also exploits label descriptors. Although zero-cnn-lwan did not outperform cnn-lwan overall on mimic-ii and mimic-iii, it had much improved results in few-shot and zero-shot learning, among other variations of zero-cnn-lwan that exploit the hierarchical relations of the labels with graph convolutions.

We note that the label-wise attention methods of Mullenbach et al. (2018) and Rios and Kavuluru (2018) were not compared to strong generic text classification baselines, such as attention-based rnns You et al. (2018) or Hierarchical Attention Networks (hans) Yang et al. (2016), which we investigate below.

3 The New Dataset

As already noted, eurlex57k contains 57k legislative documents from eur-lex 222Our dataset is available at http://nlp.cs.aueb.gr/software_and_datasets/EURLEX57K, with permission of reuse under European Union©, https://eur-lex.europa.eu, 1998–2019. with an average length of 727 words (Table 1).333See Appendix A for more statistics. Each document contains four major zones: the header, which includes the title and name of the legal body enforcing the legal act; the recitals, which are legal background references; the main body, usually organized in articles; and the attachments (e.g., appendices, annexes).

Some of the lmtc methods we consider need to be fed with documents split into smaller units. These are often sentences, but in our experiments they are sections, thus we preprocessed the raw text, respectively. We treat the header, the recitals zone, each article of the main body, and the attachments as separate sections.

All the documents of the dataset have been annotated by the Publications Office of eu 444See https://publications.europa.eu/en. with multiple concepts from eurovoc. While eurovoc includes approx. 7k concepts (labels), only 4,271 (59.31%) are present in eurlex57k, from which only 2,049 (47.97%) have been assigned to more than 10 documents. Similar distributions were reported by Rios and Kavuluru (2018) for the mimic datasets. We split eurlex57k into training (45k documents), development (6k), and test subsets (6k). We also divide the 4,271 labels into frequent (746 labels), few-shot (3,362), and zero-shot (163), depending on whether they were assigned to more than 50, fewer than 50 but at least one, or no training documents, respectively.

4 Methods

Exact Match, Logistic Regression:

A first naive baseline, Exact Match, assigns only labels whose descriptors can be found verbatim in the document. A second one uses Logistic Regression with feature vectors containing tf-idf scores of $n$-grams ($n=1,2,\dots,5$).

bigru-att:

The first neural method is a bigru with self-attention Xu et al. (2015). Each document is represented as the sequence of its word embeddings, which go through a stack of bigrus (Figure 1a). A document embedding ($h$) is computed as the sum of the resulting context-aware embeddings ($h=\sum_{i}a_{i}h_{i}$), weighted by the self-attention scores ($a_{i}$), and goes through a dense layer of $L=4,271$ output units with sigmoids, producing $L$ probabilities, one per label.

han:

The Hierarchical Attention Network Yang et al. (2016) is a strong baseline for text classification. We use a slightly modified version, where a bigru with self-attention reads the words of each section, as in bigru-att but separately per section, producing section embeddings. A second-level bigru with self-attention reads the section embeddings, producing a single document embedding ($h$) that goes through a similar output layer as in bigru-att (Figure 1b).

cnn-lwan, bigru-lwan:

In the original Label-Wise Attention Network (lwan) of Mullenbach et al. (2018), called cnn-lwan here, the word embeddings of each document are first converted to a sequence of vectors $h_{i}$ by a cnn encoder. A modified version of cnn-lwan that we developed, called bigru-lwan, replaces the cnn encoder with a bigru (Figure 1c), which converts the word embeddings into context-sensitive embeddings $h_{i}$, much as in bigru-att. Unlike bigru-att, however, both cnn-lwan and bigru-lwan use $L$ independent attention heads, one per label, generating $L$ document embeddings ($h^{(l)}=\sum_{i}a_{l,i}h_{i}$, $l=1,\dots,L$) from the sequence of vectors $h_{i}$ produced by the cnn or bigru encoder, respectively. Each document embedding ($h^{(l)}$) is specialized to predict the corresponding label and goes through a separate dense layer ($L$ dense layers in total) with a sigmoid, to produce the probability of the corresponding label.

zero-cnn-lwan, zero-bigru-lwan:

Rios and Kavuluru (2018) designed a model similar to cnn-lwan, called zacnn in their work and zero-cnn-lwan here, to deal with rare labels. In zero-cnn-lwan, the attention scores ($a_{l,i}$) and the label probabilities are produced by comparing the $h_{i}$ vectors that the cnn encoder produces and the label-specific document embeddings ($h^{(l)}$), respectively, to label embeddings. Each label embedding is the centroid of the pretrained word embeddings of the label’s descriptor; consult Rios and Kavuluru (2018) for further details. By contrast, cnn-lwan and bigru-lwan do not consider the descriptors of the labels. We also experiment with a variant of zero-cnn-lwan that we developed, dubbed zero-bigru-lwan, where the cnn encoder is replaced by a bigru.

bert:

bert (Devlin et al., 2018) is a language model based on Transformers (Vaswani et al., 2017) pretrained on large corpora. For a new target task, a task-specific layer is added on top of bert. The extra layer is trained jointly with bert by fine-tuning on task-specific data. We add a dense layer on top of bert, with sigmoids, that produces a probability per label. Unfortunately, bert can currently process texts up to 512 wordpieces, which is too small for the documents of eurlex57k. Hence, bert can only be applied to truncated versions of our documents (see below).

5 Experiments

Evaluation measures:

Common lmtc evaluation measures are precision ($P@K$) and recall ($R@K$) at the top $K$ predicted labels, averaged over test documents, micro-averaged F1 over all labels, and $nDCG@K$ Manning et al. (2009). However, $P@K$ and $R@K$ unfairly penalize methods when the gold labels of a document are fewer or more than $K$, respectively. Similar concerns have led to the introduction of $\mathrm{R}\text{-}\mathrm{Precision}$ and $nDCG@K$ in Information Retrieval Manning et al. (2009), which we believe are also more appropriate for lmtc. Note, however, that $\mathrm{R}\text{-}\mathrm{Precision}$ requires the number of gold labels per document to be known beforehand, which is unrealistic in practical applications. Therefore we propose using $\mathrm{R}\text{-}\mathrm{Precision}@K$ ($RP@K$), where $K$ is a parameter. This measure is the same as $P@K$ if there are at least $K$ gold labels, otherwise $K$ is reduced to the number of gold labels.

Figure 2 shows $RP@K$ for the three best systems, macro-averaged over test documents. Unlike $P@K$, $RP@K$ does not decline sharply as $K$ increases, because it replaces $K$ by the number of gold labels, when the latter is lower than $K$. For $K=1$, $RP@K$ is equivalent to $P@K$, as confirmed by Fig. 2. For large values of $K$ that almost always exceed the number of gold labels, $RP@K$ asymptotically approaches $R@K$, as also confirmed by Fig. 2.555See Appendix 6 for a more detailed discussion on the evaluation measures. In our dataset, there are 5.07 labels per document, hence $K=5$ is reasonable.666Evaluating at other values of $K$ lead to similar conclusions (see Fig. 2 and Appendix D).

Setup:

Hyper-parameters are tuned using the hyperopt library selecting the values with the best loss on development data.777We implemented all methods in keras (https://keras.io/). Code available at https://github.com/iliaschalkidis/lmtc-eurlex57k.git. See Appendix B for details on hyper-parameter tuning. For the best hyper-parameter values, we perform five runs and report mean scores on test data. For statistical significance tests, we take the run of each method with the best performance on development data, and perform two-tailed approximate randomization tests Dror et al. (2018) on test data.888We perform 10k iterations, randomly swapping in each iteration the responses (sets of returned labels) of the two compared systems for 50% of the test documents. Unless otherwise stated, we used 200-D pretrained glove embeddings Pennington et al. (2014).

Full documents:

The first five horizontal zones of Table 2 report results for full documents. The naive baselines are weak, as expected. Interestingly, for all, frequent, and even few-shot labels, the generic bigru-att performs better than cnn-lwan, which was designed for lmtc. han also performs better than cnn-lwan for all and frequent labels. However, replacing the cnn encoder of cnn-lwan with a bigru (bigru-lwan) leads to the best results, indicating that the main weakness of cnn-lwan is its vanilla cnn encoder.

The zero-shot versions of cnn-lwan and bigru-lwan outperform all other methods on zero-shot labels (Table 2), in line with the findings of Rios and Kavuluru (2018), because they exploit label descriptors, but more importantly because they have a component that uses prior knowledge as is (i.e., label embeddings are frozen). Exact Match also performs better on zero-shot labels, for the same reason (i.e., the prior knowledge is intact). bigru-lwan, however, is still the best method in few-shot learning. All the differences between the best (bold) and other methods in Table 2 are statistically significant ($p<0.01$).

Table 3 shows that using word2vec embeddings trained on legal texts (l2v) Chalkidis and Kampas (2018) or elmo embeddings Peters et al. (2018) trained on generic texts further improve the performance of bigru-lwan.

Document zones:

Table 4 compares the performance of bigru-lwan on the development set for different combinations of document zones (Section 3): header (H), recitals (R), main body (MB), full text. Surprisingly H+R leads to almost the same results as full documents,999The approximate randomization tests detected no statistically significant difference in this case ($p=0.20$). indicating that H+R provides most of the information needed to assign eurovoc labels.

First 512 tokens:

Given that H+R contains enough information and is shorter than 500 tokens in 83% of our dataset’s documents, we also apply bert to the first 512 tokens of each document (truncated to bert’s max. length), comparing to bigru-lwan also operating on the first 512 tokens. Table 2 (bottom zone) shows that bert outperforms all other methods, even though it considers only the first 512 tokens. It fails, however, in zero-shot learning, since it does not have a component that exploits prior knowledge as is (i.e., all the components are fine-tuned on training data).

6 Limitations and Future Work

One major limitation of the investigated methods is that they are unsuitable for Extreme Multi-Label Text Classification where there are hundreds of thousands of labels Liu et al. (2017); Zhang et al. (2018); Wydmuch et al. (2018), as opposed to the lmtc setting of our work where the labels are in the order of thousands. We leave the investigation of methods for extremely large label sets for future work. Moreover, rnn (and gru) based methods have high computational cost, especially for long documents. We plan to investigate more computationally efficient methods, e.g., dilated cnns Kalchbrenner et al. (2017) and Transformers Vaswani et al. (2017); Dai et al. (2019). We also plan to experiment with hierarchical flavors of bert to surpass its length limitations. Furthermore, experimenting with more datasets e.g., rcv1, Amazon-13K, Wiki-30K, mimic-iii will allow us to confirm our conclusions in different domains. Finally, we plan to investigate Generalized Zero-Shot Learning Liu et al. (2018).

Acknowledgements

This work was partly supported by the Research Center of the Athens University of Economics and Business.

Appendix

Appendix A eurlex57k statistics

Figure 3 shows the distribution of labels across eurlex57k documents. From the 7k labels fewer than 50% appear in more than 10 documents. Such an aggressive Zipfian distribution has also been noted in medical code predictions Rios and Kavuluru (2018), where such thesauri are used to classify documents, demonstrating the practical importance of few-shot and zero-shot learning.

Appendix B Hyper-paramater tuning

Table 5 shows the best hyper-parameters returned by hyperopt. Concerning bert, we set the dropout rate and learning rate to 0.1 and 5e-5, respectively, as suggested by Devlin et al. (2018), while batch size was set to 8 due to gpu memory limitations. Finally, we noticed that the model did not converge in the fourth epoch, as suggested by Devlin et al. (2018). Thus we used early-stopping with no patience and trained the model for eight to nine epochs on average among the five runs.

Appendix C Evaluation Measures

The macro-averaged versions of $R@K$ and $P@K$ are defined as follows:

[TABLE]

where $T$ is the total number of test documents, $K$ is the number of labels to be selected per document, $S_{t}(K)$ is the number of correct labels among those ranked as top $K$ for the $t$-th document, and $R_{t}$ is the number of gold labels for each document. Although these measures are widely used in lmtc, we question their appropriateness for the following reasons:

$R@K$ leads to excessive penalization when documents have more than $K$ gold labels. For example, evaluating at $K=1$ for a single document with 5 gold labels returns $R@1=0.20$, if the system managed to return a correct label. The system is penalized, even though it was not allowed to return more than one label. 2. 2.

$P@K$ does the same for documents with fewer than $K$ gold labels. For example, evaluating at $K=5$ for a single document with a single gold label returns $P@1=0.20$. 3. 3.

Both measures over- or under-estimate performance on documents whose number of gold labels largely diverges from $K$. This is clearly illustrated in Figure 2 of the main article. 4. 4.

Because of these drawbacks, both measures do not correctly single out the best methods.

Based on the above arguments, we believe that R-Precision@K ($RP@K$) and $nDCG@K$ lead to a more informative and fair evaluation. Both measures adjust to the number of gold labels per document, without over- or under-estimating performance when documents have few or many gold labels. The macro-averaged versions of the two measures are defined as follows:

[TABLE]

Again, $T$ is the total number of test documents, $K$ is the number of labels to be selected, $S_{t}(K)$ is the number of correct labels among those ranked as top $K$ for the $t$-th document, and $R_{t}$ is the number of gold labels for each document. In the main article we report results for $K=5$. The reason is that the majority of the documents of eurlex57k (57.7%) have at most 5 labels. The detailed distributions can be seen in Figure 4.

Appendix D Experimental Results

In Tables 6–9, we present additional results for the main measures used across the lmtc literature ($P@K$, $R@K$, $RP@K$, $nDGC@K$).