FlexiTerm: A more efficient implementation of flexible multi-word term recognition

TL;DR

FlexiTerm is an unsupervised, scalable Python implementation that significantly improves the efficiency of recognizing multi-word terms in domain-specific texts, enabling practical large-scale applications.

Contribution

The paper presents a re-implementation of FlexiTerm in Python, achieving major efficiency improvements over the Java version for large-scale multi-word term recognition.

Findings

Python implementation is more efficient than Java

FlexiTerm can handle larger corpora effectively

Transition from proof of concept to production-grade system

Abstract

Terms are linguistic signifiers of domain-specific concepts. Automated recognition of multi-word terms (MWT) in free text is a sequence labelling problem, which is commonly addressed using supervised machine learning methods. Their need for manual annotation of training data makes it difficult to port such methods across domains. FlexiTerm, on the other hand, is a fully unsupervised method for MWT recognition from domain-specific corpora. Originally implemented in Java as a proof of concept, it did not scale well, thus offering little practical value in the context of big data. In this paper, we describe its re-implementation in Python and compare the performance of these two implementations. The results demonstrated major improvements in terms of efficiency, which allow FlexiTerm to transition from the proof of concept to the production-grade application.