On the Choice of Auxiliary Languages for Improved Sequence Tagging

TL;DR

This paper investigates how the choice of auxiliary languages affects sequence tagging performance, revealing that relatedness isn't always predictive of effectiveness, and introduces attention-based meta-embeddings that achieve state-of-the-art results.

Contribution

It demonstrates that language relatedness alone doesn't determine auxiliary language effectiveness and proposes attention-based meta-embeddings for improved sequence tagging.

Findings

Relatedness isn't always predictive of auxiliary language effectiveness.

Attention-based meta-embeddings outperform previous methods.

State-of-the-art POS tagging results achieved in five languages.

Abstract

Recent work showed that embeddings from related languages can improve the performance of sequence tagging, even for monolingual models. In this analysis paper, we investigate whether the best auxiliary language can be predicted based on language distances and show that the most related language is not always the best auxiliary language. Further, we show that attention-based meta-embeddings can effectively combine pre-trained embeddings from different languages for sequence tagging and set new state-of-the-art results for part-of-speech tagging in five languages.