Investigating Task Arithmetic for Zero-Shot Information Retrieval

TL;DR

This paper explores Task Arithmetic, a simple weight combination technique for LLMs, to improve zero-shot information retrieval across diverse datasets without additional fine-tuning.

Contribution

It introduces Task Arithmetic as a novel method for zero-shot model adaptation, enabling effective retrieval performance across multiple domains.

Findings

Up to 18% improvement in NDCG@10

Up to 15% improvement in P@10

Effective synthesis of diverse tasks and domains

Abstract

Large Language Models (LLMs) have shown impressive zero-shot performance across a variety of Natural Language Processing tasks, including document re-ranking. However, their effectiveness degrades on unseen tasks and domains, largely due to shifts in vocabulary and word distributions. In this paper, we investigate Task Arithmetic, a technique that combines the weights of LLMs pre-trained on different tasks or domains via simple mathematical operations, such as addition or subtraction, to adapt retrieval models without requiring additional fine-tuning. Our method is able to synthesize diverse tasks and domain knowledge into a single model, enabling effective zero-shot adaptation in different retrieval contexts. Extensive experiments on publicly available scientific, biomedical, and multilingual datasets show that our method improves state-of-the-art re-ranking performance by up to 18% in NDCG@10 and 15% in P@10. In addition to these empirical gains, our analysis provides insights into the strengths and limitations of Task Arithmetic as a practical strategy for zero-shot learning and model adaptation. We make our code publicly available at https://github.com/DetectiveMB/Task-Arithmetic-for-ZS-IR.