Fishing for Exactness

TL;DR

This paper advocates using Fisher's exact test over traditional asymptotic tests for identifying dependent word pairs in natural language, demonstrating its superior reliability especially with sparse and skewed data.

Contribution

It introduces Fisher's exact test as a more appropriate statistical method for dependency detection in NLP, supported by theoretical and experimental comparisons.

Findings

Fisher's exact test outperforms asymptotic tests in dependency detection.

Fisher's test is more reliable with sparse, skewed data.

The approach is applicable to various NLP problems with similar data characteristics.

Abstract

Statistical methods for automatically identifying dependent word pairs (i.e. dependent bigrams) in a corpus of natural language text have traditionally been performed using asymptotic tests of significance. This paper suggests that Fisher's exact test is a more appropriate test due to the skewed and sparse data samples typical of this problem. Both theoretical and experimental comparisons between Fisher's exact test and a variety of asymptotic tests (the t-test, Pearson's chi-square test, and Likelihood-ratio chi-square test) are presented. These comparisons show that Fisher's exact test is more reliable in identifying dependent word pairs. The usefulness of Fisher's exact test extends to other problems in statistical natural language processing as skewed and sparse data appears to be the rule in natural language. The experiment presented in this paper was performed using PROC FREQ of the SAS System.