BiCA: Effective Biomedical Dense Retrieval with Citation-Aware Hard Negatives

TL;DR

This paper introduces BiCA, a biomedical dense retrieval method that leverages citation links to generate hard negatives, significantly improving retrieval performance in biomedical and scientific domains with minimal fine-tuning.

Contribution

BiCA utilizes citation-aware hard negatives from PubMed to enhance domain-specific dense retrieval models, demonstrating state-of-the-art results with minimal data and fine-tuning.

Findings

Improved zero-shot dense retrieval performance on BEIR and LoTTE datasets.

Citation-informed negatives lead to better domain adaptation and retrieval accuracy.

State-of-the-art results with minimal fine-tuning in biomedical retrieval tasks.

Abstract

Hard negatives are essential for training effective retrieval models. Hard-negative mining typically relies on ranking documents using cross-encoders or static embedding models based on similarity metrics such as cosine distance. Hard negative mining becomes challenging for biomedical and scientific domains due to the difficulty in distinguishing between source and hard negative documents. However, referenced documents naturally share contextual relevance with the source document but are not duplicates, making them well-suited as hard negatives. In this work, we propose BiCA: Biomedical Dense Retrieval with Citation-Aware Hard Negatives, an approach for hard-negative mining by utilizing citation links in 20,000 PubMed articles for improving a domain-specific small dense retriever. We fine-tune the GTE_small and GTE_Base models using these citation-informed negatives and observe consistent improvements in zero-shot dense retrieval using nDCG@10 for both in-domain and out-of-domain tasks on BEIR and outperform baselines on long-tailed topics in LoTTE using Success@5. Our findings highlight the potential of leveraging document link structure to generate highly informative negatives, enabling state-of-the-art performance with minimal fine-tuning and demonstrating a path towards highly data-efficient domain adaptation.