Ensembling Transformers for Cross-domain Automatic Term Extraction

TL;DR

This study evaluates the effectiveness of Transformer-based pretrained language models for automatic term extraction across multiple languages and domains, demonstrating that monolingual models generally outperform multilingual ones, with ensemble methods further enhancing performance.

Contribution

It provides a comprehensive comparison of monolingual and multilingual Transformer models for cross-domain, multi-language term extraction and introduces ensemble strategies that improve extraction accuracy.

Findings

Monolingual models outperform multilingual models in most cases.

Ensemble of top models yields significant performance improvements.

Performance varies depending on language and whether named entity terms are included.

Abstract

Automatic term extraction plays an essential role in domain language understanding and several natural language processing downstream tasks. In this paper, we propose a comparative study on the predictive power of Transformers-based pretrained language models toward term extraction in a multi-language cross-domain setting. Besides evaluating the ability of monolingual models to extract single- and multi-word terms, we also experiment with ensembles of mono- and multilingual models by conducting the intersection or union on the term output sets of different language models. Our experiments have been conducted on the ACTER corpus covering four specialized domains (Corruption, Wind energy, Equitation, and Heart failure) and three languages (English, French, and Dutch), and on the RSDO5 Slovenian corpus covering four additional domains (Biomechanics, Chemistry, Veterinary, and Linguistics). The results show that the strategy of employing monolingual models outperforms the state-of-the-art approaches from the related work leveraging multilingual models, regarding all the languages except Dutch and French if the term extraction task excludes the extraction of named entity terms. Furthermore, by combining the outputs of the two best performing models, we achieve significant improvements.