Priberam Labs at the NTCIR-15 SHINRA2020-ML: Classification Task

TL;DR

This paper presents models based on Multilingual BERT to classify Wikipedia entities into a large hierarchical ontology, achieving effective multilingual and zero-shot language classification for knowledge structuring.

Contribution

The work introduces three BERT-based models with various pooling and hierarchy strategies for entity classification across 268 categories.

Findings

High classification accuracy across multiple languages

Effective zero-shot language classification

Comparison of different pooling and hierarchy strategies

Abstract

Wikipedia is an online encyclopedia available in 285 languages. It composes an extremely relevant Knowledge Base (KB), which could be leveraged by automatic systems for several purposes. However, the structure and organisation of such information are not prone to automatic parsing and understanding and it is, therefore, necessary to structure this knowledge. The goal of the current SHINRA2020-ML task is to leverage Wikipedia pages in order to categorise their corresponding entities across 268 hierarchical categories, belonging to the Extended Named Entity (ENE) ontology. In this work, we propose three distinct models based on the contextualised embeddings yielded by Multilingual BERT. We explore the performances of a linear layer with and without explicit usage of the ontology's hierarchy, and a Gated Recurrent Units (GRU) layer. We also test several pooling strategies to leverage BERT's embeddings and selection criteria based on the labels' scores. We were able to achieve good performance across a large variety of languages, including those not seen during the fine-tuning process (zero-shot languages).