On the Estimation and Use of Statistical Modelling in Information Retrieval

TL;DR

This paper advocates for a statistically principled approach to determine the true data distribution in information retrieval, replacing assumptions with data-driven models that improve retrieval effectiveness.

Contribution

It introduces a new method for identifying the true distribution in IR data and develops adaptive ranking models based on this approach.

Findings

Achieves comparable or better results than strong baselines on TREC datasets.

Demonstrates the effectiveness of data-driven distribution estimation in IR.

Shows improved retrieval performance using the proposed models.

Abstract

Several tasks in information retrieval (IR) rely on assumptions regarding the distribution of some property (such as term frequency) in the data being processed. This thesis argues that such distributional assumptions can lead to incorrect conclusions and proposes a statistically principled method for determining the "true" distribution. This thesis further applies this method to derive a new family of ranking models that adapt their computations to the statistics of the data being processed. Experimental evaluation shows results on par or better than multiple strong baselines on several TREC collections. Overall, this thesis concludes that distributional assumptions can be replaced with an effective, efficient and principled method for determining the "true" distribution and that using the "true" distribution can lead to improved retrieval performance.