A Study of MatchPyramid Models on Ad-hoc Retrieval

TL;DR

This paper evaluates the MatchPyramid deep neural network model for ad-hoc retrieval, demonstrating its strengths and limitations compared to traditional retrieval methods through extensive experiments.

Contribution

It is the first comprehensive study of MatchPyramid on ad-hoc retrieval, analyzing various model configurations and comparing its performance with traditional methods.

Findings

MatchPyramid outperforms recent deep matching models in retrieval tasks.

Traditional models like BM25 still outperform MatchPyramid in effectiveness.

Model performance is sensitive to pooling sizes, interaction functions, and kernel sizes.

Abstract

Deep neural networks have been successfully applied to many text matching tasks, such as paraphrase identification, question answering, and machine translation. Although ad-hoc retrieval can also be formalized as a text matching task, few deep models have been tested on it. In this paper, we study a state-of-the-art deep matching model, namely MatchPyramid, on the ad-hoc retrieval task. The MatchPyramid model employs a convolutional neural network over the interactions between query and document to produce the matching score. We conducted extensive experiments to study the impact of different pooling sizes, interaction functions and kernel sizes on the retrieval performance. Finally, we show that the MatchPyramid models can significantly outperform several recently introduced deep matching models on the retrieval task, but still cannot compete with the traditional retrieval models, such as BM25 and language models.