Improving Multitask Retrieval by Promoting Task Specialization

TL;DR

This paper introduces a method to improve multitask retrieval by promoting task specialization, using a specialized pretrained model, prompting, and adaptive learning, resulting in better performance than naive approaches.

Contribution

It presents a novel adaptive learning technique and model design that enhances task specialization in multitask retrieval, surpassing task-specific retrievers.

Findings

Outperforms task-specific retrievers on KILT benchmark

Parameters become more task-specialized with the proposed method

Adaptive learning encourages parameter specialization in multitask models

Abstract

In multitask retrieval, a single retriever is trained to retrieve relevant contexts for multiple tasks. Despite its practical appeal, naive multitask retrieval lags behind task-specific retrieval in which a separate retriever is trained for each task. We show that it is possible to train a multitask retriever that outperforms task-specific retrievers by promoting task specialization. The main ingredients are: (1) a better choice of pretrained model (one that is explicitly optimized for multitasking) along with compatible prompting, and (2) a novel adaptive learning method that encourages each parameter to specialize in a particular task. The resulting multitask retriever is highly performant on the KILT benchmark. Upon analysis, we find that the model indeed learns parameters that are more task-specialized compared to naive multitasking without prompting or adaptive learning.