Probability Ranking in Vector Spaces

TL;DR

This paper introduces a novel vector space separation method based on the Probability Ranking Principle, demonstrating improved retrieval performance through mathematical proof and experimental validation.

Contribution

It proposes a new vector space separation technique that outperforms traditional subset separation methods in information retrieval.

Findings

Mathematically proven superiority of the vector space separation method

Experimental results show improved recall and fallout performance

Theoretical framework extends the Probability Ranking Principle

Abstract

The Probability Ranking Principle states that the document set with the highest values of probability of relevance optimizes information retrieval effectiveness given the probabilities are estimated as accurately as possible. The key point of the principle is the separation of the document set into two subsets with a given level of fallout and with the highest recall. The paper introduces the separation between two vector subspaces and shows that the separation yields a more effective performance than the optimal separation into subsets with the same available evidence, the performance being measured with recall and fallout. The result is proved mathematically and exemplified experimentally.