Determining the Unithood of Word Sequences using a Probabilistic Approach

TL;DR

This paper introduces a new probabilistic measure for assessing the unithood of word sequences, independent of termhood, utilizing linguistic and statistical evidence, and demonstrates improved accuracy over existing methods.

Contribution

A novel probabilistic approach for unithood determination that combines linguistic parsing and Google search data, outperforming previous empirical methods.

Findings

Improved precision, recall, and accuracy in unithood measurement.

Effective use of Google search engine data for statistical evidence.

Validation on 1,825 test cases showing superiority over existing methods.

Abstract

Most research related to unithood were conducted as part of a larger effort for the determination of termhood. Consequently, novelties are rare in this small sub-field of term extraction. In addition, existing work were mostly empirically motivated and derived. We propose a new probabilistically-derived measure, independent of any influences of termhood, that provides dedicated measures to gather linguistic evidence from parsed text and statistical evidence from Google search engine for the measurement of unithood. Our comparative study using 1,825 test cases against an existing empirically-derived function revealed an improvement in terms of precision, recall and accuracy.