Multilingual Hierarchical Attention Networks for Document Classification

TL;DR

This paper introduces multilingual hierarchical attention networks that share components across languages, enabling effective document classification with fewer parameters and improved transfer, demonstrated on a large multilingual news dataset.

Contribution

The paper proposes a novel multilingual hierarchical attention network architecture with shared encoders and attention mechanisms, enhancing cross-language transfer and efficiency.

Findings

Multilingual models outperform monolingual models in low-resource settings.

Shared models use fewer parameters than separate models.

Models achieve high accuracy on a large multilingual news dataset.

Abstract

Hierarchical attention networks have recently achieved remarkable performance for document classification in a given language. However, when multilingual document collections are considered, training such models separately for each language entails linear parameter growth and lack of cross-language transfer. Learning a single multilingual model with fewer parameters is therefore a challenging but potentially beneficial objective. To this end, we propose multilingual hierarchical attention networks for learning document structures, with shared encoders and/or shared attention mechanisms across languages, using multi-task learning and an aligned semantic space as input. We evaluate the proposed models on multilingual document classification with disjoint label sets, on a large dataset which we provide, with 600k news documents in 8 languages, and 5k labels. The multilingual models outperform monolingual ones in low-resource as well as full-resource settings, and use fewer parameters, thus confirming their computational efficiency and the utility of cross-language transfer.