Getting Beyond the State of the Art of Information Retrieval with Quantum Theory

TL;DR

This paper introduces a quantum theory-based approach to information retrieval, demonstrating mathematically and experimentally that vector subspace separation outperforms classical subset separation in effectiveness.

Contribution

It presents a novel quantum-inspired vector probability model for information retrieval, showing its superiority over traditional methods.

Findings

Vector subspace separation improves retrieval effectiveness.

Mathematical proof of quantum theory's advantage in IR.

Experimental validation confirms the theoretical results.

Abstract

According to the probability ranking principle, the document set with the highest values of probability of relevance optimizes information retrieval effectiveness given the probabilities are estimated as accurately as possible. The key point of this principle is the separation of the document set into two subsets with a given level of fallout and with the highest recall. If subsets of set measures are replaced by subspaces and space measures, we obtain an alternative theory stemming from Quantum Theory. That theory is named after vector probability because vectors represent event like sets do in classical probability. The paper shows that the separation into vector subspaces is more effective than the separation into subsets with the same available evidence. The result is proved mathematically and verified experimentally. In general, the paper suggests that quantum theory is not only a source of rhetoric inspiration, but is a sufficient condition to improve retrieval effectiveness in a principled way.