Scalable Methods for Calculating Term Co-Occurrence Frequencies

TL;DR

This paper introduces scalable, exact methods for computing term co-occurrence frequencies in large document collections, significantly improving speed over naive approaches by using inverted indexes and novel list-pairs techniques.

Contribution

The paper presents a new list-pairs method that enables fast, exact calculation of term co-occurrence frequencies without approximation, outperforming traditional dictionary-based approaches.

Findings

Naive dictionary approach is very slow.

Inverted indexes combined with linear scanning greatly improve speed.

The list-pairs method processes hundreds of thousands of documents per hour.

Abstract

Search techniques make use of elementary information such as term frequencies and document lengths in computation of similarity weighting. They can also exploit richer statistics, in particular the number of documents in which any two terms co-occur. In this paper we propose alternative methods for computing this statistic, a challenging task because the number of distinct pairs of terms is vast -- around 100,000 in a typical 1000-word news article, for example. In contrast, we do not employ approximation algorithms, as we want to be able to find exact counts. We explore their efficiency, finding that a na\"ive approach based on a dictionary is indeed very slow, while methods based on a combination of inverted indexes and linear scanning provide both massive speed-ups and better observed asymptotic behaviour. Our careful implementation shows that, with our novel list-pairs approach it is possible to process over several hundred thousand documents per hour.