Neural Methods for Effective, Efficient, and Exposure-Aware Information Retrieval

TL;DR

This paper introduces novel neural architectures tailored for information retrieval, addressing challenges like vocabulary mismatch, efficiency in large collections, and exposure control, to improve relevance and fairness.

Contribution

It proposes new neural models specifically designed for IR, tackling vocabulary mismatch, efficiency, and exposure fairness in large-scale retrieval systems.

Findings

Improved relevance ranking in IR tasks

Enhanced efficiency for large document collections

Incorporation of exposure fairness mechanisms

Abstract

Neural networks with deep architectures have demonstrated significant performance improvements in computer vision, speech recognition, and natural language processing. The challenges in information retrieval (IR), however, are different from these other application areas. A common form of IR involves ranking of documents--or short passages--in response to keyword-based queries. Effective IR systems must deal with query-document vocabulary mismatch problem, by modeling relationships between different query and document terms and how they indicate relevance. Models should also consider lexical matches when the query contains rare terms--such as a person's name or a product model number--not seen during training, and to avoid retrieving semantically related but irrelevant results. In many real-life IR tasks, the retrieval involves extremely large collections--such as the document index of a commercial Web search engine--containing billions of documents. Efficient IR methods should take advantage of specialized IR data structures, such as inverted index, to efficiently retrieve from large collections. Given an information need, the IR system also mediates how much exposure an information artifact receives by deciding whether it should be displayed, and where it should be positioned, among other results. Exposure-aware IR systems may optimize for additional objectives, besides relevance, such as parity of exposure for retrieved items and content publishers. In this thesis, we present novel neural architectures and methods motivated by the specific needs and challenges of IR tasks.