Deep Learning Based Dense Retrieval: A Comparative Study

TL;DR

This paper compares the robustness of various dense retrieval models against tokenizer poisoning, revealing that supervised models are more vulnerable than unsupervised ones, and small perturbations can greatly reduce retrieval accuracy.

Contribution

It provides a systematic evaluation of dense retrievers' vulnerability to tokenizer poisoning, highlighting the need for developing more robust models.

Findings

Supervised models like BERT and DPR are highly vulnerable to tokenizer poisoning.

Unsupervised models like ANCE demonstrate greater resilience.

Small perturbations can significantly impair retrieval performance.

Abstract

Dense retrievers have achieved state-of-the-art performance in various information retrieval tasks, but their robustness against tokenizer poisoning remains underexplored. In this work, we assess the vulnerability of dense retrieval systems to poisoned tokenizers by evaluating models such as BERT, Dense Passage Retrieval (DPR), Contriever, SimCSE, and ANCE. We find that supervised models like BERT and DPR experience significant performance degradation when tokenizers are compromised, while unsupervised models like ANCE show greater resilience. Our experiments reveal that even small perturbations can severely impact retrieval accuracy, highlighting the need for robust defenses in critical applications.