Feature Selective Likelihood Ratio Estimator for Low- and Zero-frequency N-grams

TL;DR

This paper introduces a feature selective likelihood ratio estimator for low- and zero-frequency N-grams in NLP, combining feature selection with likelihood ratio estimation to improve efficiency and accuracy.

Contribution

It proposes a novel method that integrates feature selection into likelihood ratio estimation for rare N-grams, addressing efficiency and accuracy issues.

Findings

Effective estimation for low- and zero-frequency N-grams

Reduced computational time and memory usage

Improved accuracy over traditional methods

Abstract

In natural language processing (NLP), the likelihood ratios (LRs) of N-grams are often estimated from the frequency information. However, a corpus contains only a fraction of the possible N-grams, and most of them occur infrequently. Hence, we desire an LR estimator for low- and zero-frequency N-grams. One way to achieve this is to decompose the N-grams into discrete values, such as letters and words, and take the product of the LRs for the values. However, because this method deals with a large number of discrete values, the running time and memory usage for estimation are problematic. Moreover, use of unnecessary discrete values causes deterioration of the estimation accuracy. Therefore, this paper proposes combining the aforementioned method with the feature selection method used in document classification, and shows that our estimator provides effective and efficient estimation results for low- and zero-frequency N-grams.