Remedies against the Vocabulary Gap in Information Retrieval

TL;DR

This paper addresses the vocabulary gap in information retrieval caused by different word choices in queries and documents, proposing methods to improve query formulation and latent vector models to enhance relevance matching.

Contribution

It introduces novel methods for effective query formulation from complex texts and latent vector space models to mitigate the vocabulary gap in IR.

Findings

Improved relevance ranking with new query formulation techniques.

Latent vector models better capture semantic similarity beyond exact term matches.

Reduction in the impact of vocabulary mismatch in retrieval results.

Abstract

Search engines rely heavily on term-based approaches that represent queries and documents as bags of words. Text---a document or a query---is represented by a bag of its words that ignores grammar and word order, but retains word frequency counts. When presented with a search query, the engine then ranks documents according to their relevance scores by computing, among other things, the matching degrees between query and document terms. While term-based approaches are intuitive and effective in practice, they are based on the hypothesis that documents that exactly contain the query terms are highly relevant regardless of query semantics. Inversely, term-based approaches assume documents that do not contain query terms as irrelevant. However, it is known that a high matching degree at the term level does not necessarily mean high relevance and, vice versa, documents that match null query terms may still be relevant. Consequently, there exists a vocabulary gap between queries and documents that occurs when both use different words to describe the same concepts. It is the alleviation of the effect brought forward by this vocabulary gap that is the topic of this dissertation. More specifically, we propose (1) methods to formulate an effective query from complex textual structures and (2) latent vector space models that circumvent the vocabulary gap in information retrieval.