Improving Multitask Retrieval by Promoting Task Specialization
Wenzheng Zhang, Chenyan Xiong, Karl Stratos, Arnold Overwijk

TL;DR
This paper introduces a method to improve multitask retrieval by promoting task specialization, using a specialized pretrained model, prompting, and adaptive learning, resulting in better performance than naive approaches.
Contribution
It presents a novel adaptive learning technique and model design that enhances task specialization in multitask retrieval, surpassing task-specific retrievers.
Findings
Outperforms task-specific retrievers on KILT benchmark
Parameters become more task-specialized with the proposed method
Adaptive learning encourages parameter specialization in multitask models
Abstract
In multitask retrieval, a single retriever is trained to retrieve relevant contexts for multiple tasks. Despite its practical appeal, naive multitask retrieval lags behind task-specific retrieval in which a separate retriever is trained for each task. We show that it is possible to train a multitask retriever that outperforms task-specific retrievers by promoting task specialization. The main ingredients are: (1) a better choice of pretrained model (one that is explicitly optimized for multitasking) along with compatible prompting, and (2) a novel adaptive learning method that encourages each parameter to specialize in a particular task. The resulting multitask retriever is highly performant on the KILT benchmark. Upon analysis, we find that the model indeed learns parameters that are more task-specialized compared to naive multitasking without prompting or adaptive learning.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDomain Adaptation and Few-Shot Learning · Information Retrieval and Search Behavior · Multimodal Machine Learning Applications
