Consistency and Variation in Kernel Neural Ranking Model

TL;DR

This paper investigates the consistency of the K-NRM neural ranking model, revealing low variance in overall performance but significant variability in individual query rankings due to different latent matching patterns, and proposes ensemble methods to enhance effectiveness.

Contribution

It uncovers the source of ranking variability in K-NRM and introduces ensemble techniques leveraging multiple matching patterns for improved IR performance.

Findings

K-NRM shows low variance in relevance metrics across trials.

Different latent matching patterns lead to varied individual query rankings.

Ensemble rankers based on these patterns improve effectiveness and generalization.

Abstract

This paper studies the consistency of the kernel-based neural ranking model K-NRM, a recent state-of-the-art neural IR model, which is important for reproducible research and deployment in the industry. We find that K-NRM has low variance on relevance-based metrics across experimental trials. In spite of this low variance in overall performance, different trials produce different document rankings for individual queries. The main source of variance in our experiments was found to be different latent matching patterns captured by K-NRM. In the IR-customized word embeddings learned by K-NRM, the query-document word pairs follow two different matching patterns that are equally effective, but align word pairs differently in the embedding space. The different latent matching patterns enable a simple yet effective approach to construct ensemble rankers, which improve K-NRM's effectiveness and generalization abilities.