Modelling Word Burstiness in Natural Language: A Generalised Polya Process for Document Language Models in Information Retrieval

TL;DR

This paper presents a generalized Polya process for document language modeling that captures term burstiness and improves retrieval effectiveness in information retrieval systems.

Contribution

It introduces a flexible multivariate Polya process framework that unifies and extends existing language models for better modeling of document term distributions.

Findings

The proposed model effectively captures term burstiness.

Experimental results show significant improvement in retrieval performance.

The framework generalizes multiple existing language models.

Abstract

We introduce a generalised multivariate Polya process for document language modelling. The framework outlined here generalises a number of statistical language models used in information retrieval for modelling document generation. In particular, we show that the choice of replacement matrix M ultimately defines the type of random process and therefore defines a particular type of document language model. We show that a particular variant of the general model is useful for modelling term-specific burstiness. Furthermore, via experimentation we show that this variant significantly improves retrieval effectiveness over a strong baseline on a number of small test collections.