Hypencoder: Hypernetworks for Information Retrieval

TL;DR

Hypencoder introduces a novel retrieval paradigm using hypernetworks to generate query-specific relevance functions, significantly improving retrieval performance and efficiency over traditional vector-based methods.

Contribution

This paper presents Hypencoders, a new class of models that use hypernetworks to produce query-specific relevance functions, surpassing existing dense retrieval and reranking models.

Findings

Hypencoders outperform strong dense retrieval models.

Hypencoders surpass reranking models with fewer parameters.

Efficient retrieval from 8.8M documents in under 60ms.

Abstract

Existing information retrieval systems are largely constrained by their reliance on vector inner products to assess query-document relevance, which naturally limits the expressiveness of the relevance score they can produce. We propose a new paradigm; instead of representing a query as a vector, we use a small neural network that acts as a learned query-specific relevance function. This small neural network takes a document representation as input (in this work we use a single vector) and produces a scalar relevance score. To produce the small neural network we use a hypernetwork, a network that produces the weights of other networks, as our query encoder. We name this category of encoder models Hypencoders. Experiments on in-domain search tasks show that Hypencoders significantly outperform strong dense retrieval models and even surpass reranking models and retrieval models with an order of magnitude more parameters. To assess the extent of Hypencoders' capabilities, we evaluate on a set of hard retrieval tasks including tip-of-the-tongue and instruction-following retrieval tasks. On harder tasks, we find that the performance gap widens substantially compared to standard retrieval tasks. Furthermore, to demonstrate the practicality of our method, we implement an approximate search algorithm and show that our model is able to retrieve from a corpus of 8.8M documents in under 60 milliseconds.